Data Writer

Overview

Build a writer that accepts records from many threads and writes them out efficiently. Instead of touching the disk on every record, it buffers records and flushes them in batches — when the buffer fills, when a time interval elapses, or when the caller explicitly flushes. This trades a little latency for much higher throughput.

The design balances batching against durability. Bigger batches mean fewer, cheaper disk writes, but a record sitting in the buffer is not yet safe. The writer must define when a record is durable, flush on both a size and a time trigger so records don't linger, and guarantee everything is written on shutdown.

Interfaces

class DataWriter:
    def __init__(self, path, max_batch=1000, flush_interval=1.0):
        self._buffer = []
        self._lock = Lock()
        self._file = open(path, "ab")
        # a background thread flushes every flush_interval seconds

    def write(self, record):
        """Append a record; flush automatically when the batch is full."""
        with self._lock:
            self._buffer.append(record)
            if len(self._buffer) >= self._max_batch:
                self._flush_locked()

    def flush(self):
        """Force all buffered records to disk now."""
        with self._lock:
            self._flush_locked()

    def _flush_locked(self):
        if not self._buffer:
            return
        for rec in self._buffer:
            self._file.write(encode(rec))
        self._file.flush()
        os.fsync(self._file.fileno())     # make the batch durable
        self._buffer.clear()

    def close(self):
        """Flush everything and stop the background flusher."""
        ...

Records accumulate in a buffer under a lock. A background thread flushes on a timer so a trickle of records still gets persisted promptly, and write flushes early whenever the buffer hits max_batch.

Inputs and outputs

• Input: write(record) calls from many threads.
• Output: all records land in the file in batches; after a successful flush (or close), the records are durable on disk.

Requirements

• write is safe to call concurrently from many threads.
• The buffer flushes on both a size threshold and a time interval.
• A flush makes the whole batch durable (flush + fsync).
• close flushes any remaining buffered records — nothing is lost on shutdown.
• Records keep their order within the output.

Examples

w = DataWriter("events.dat", max_batch=1000, flush_interval=1.0)

for e in stream:        # called from many producer threads
    w.write(e)          # buffered; flushed in batches of 1000 or every 1s

w.close()               # final flush — everything is on disk

Follow-ups

• How would you roll the output into a new file by size or time without tearing a record across the boundary (log rotation)?

• How would you partition records into multiple files by key so downstream readers can work in parallel?

• How would a durable mode where write blocks until its batch is fsynced trade latency for safety?

• And more...