Durable Log Writer

Overview

Implement a write(record) component that many threads call at the same time. The call must not return until the record is durably persisted — safe across a process crash and a machine restart, not just sitting in a memory buffer. Two records must never be corrupted or interleaved on disk.

The real shape of this problem is many concurrent producers feeding one durable sink with strict acknowledgement semantics. The same pattern shows up as an event writer, a transaction journal, an audit trail, or a durable message-queue producer. The points being tested are concurrency correctness, synchronization, durability semantics, and where the performance bottleneck lives.

Interfaces

A single-writer design: callers hand records to a durable queue and block until the writer confirms the record hit the disk.

class LogWriter:
    def __init__(self, path):
        self._queue = Queue()              # producers -> single writer
        self._file = open(path, "ab")
        # start one background writer thread running _write_loop

    def write(self, record):
        """Block until `record` is fsynced to disk; raise if the write fails."""
        req = LogRequest(record)
        self._queue.put(req)
        req.wait_until_done()              # parks on a condition variable
        if req.error:
            raise req.error

    def _write_loop(self):
        while True:
            req = self._queue.get()
            try:
                self._file.write(encode(req.record))
                self._file.flush()         # push buffer to the OS
                os.fsync(self._file.fileno())   # force it onto the disk
                req.mark_persisted()       # notify the waiting caller
            except Exception as e:
                req.fail(e)

One dedicated writer owns the file, so records are appended in a single, non-interleaved stream. Each caller waits on its request's condition variable and is woken only once its bytes are durable.

Inputs and outputs

• Input: records arriving from any number of producer threads.
• Output: write returns only after the record is on disk; if the disk write fails, the caller gets an exception rather than a false success.

Requirements

• write is safe to call concurrently from many threads.
• A record is acknowledged only after flush + fsync — crash- and restart-safe.
• Records are never torn or interleaved on disk.
• Decide the ordering contract up front: either a global order (A acked before B appears before B) or durability-only with no cross-thread order guarantee.
• Disk failures surface as errors; the writer does not fail silently.

Examples

w = LogWriter("trades.log")

# called from many threads at once
w.write("BUY AAPL 100")    # returns only after this line is fsynced
w.write("SELL TSLA 50")    # never interleaves with the line above

On disk you must see whole records, never a torn mix like BUY SEAPL / LL TSLA.

Follow-ups

• How would you implement group commit — how large should a batch grow, or how long should it linger, before you flush?

• The file can't grow forever. How would you roll it over by size without losing or tearing a record?

• How would an async write that returns a future change the durability contract you give callers?

• And more...