Skip to content

Compaction back-pressure#

2024-11-14

When compaction can't keep up with writes, LSMs need to slow down incoming writes or they blow up. RocksDB, Cassandra, and Postgres all handle this differently.

RocksDB's write stalling#

RocksDB has several triggers, each of which stalls writes progressively harder:

  • level0_slowdown_writes_trigger (default 20): rate-limit writes.
  • level0_stop_writes_trigger (default 36): block writes entirely until L0 drains.
  • soft_pending_compaction_bytes_limit (default 64 GB): rate-limit if the amount of data waiting to compact exceeds this.
  • hard_pending_compaction_bytes_limit (default 256 GB): block writes.

The soft stall applies a delay (via RateLimiter) proportional to how deep you are into the stall region. The hard stop simply blocks Put() until compaction catches up.

Cassandra's back-pressure#

Cassandra doesn't stall clients directly. Instead:

  • The commit log grows and the memtable count climbs.
  • Once memtable_flush_writers are all busy, new writes have to wait for a slot.
  • Under sustained overload, JVM heap pressure causes GC pauses, which act as organic back-pressure.

This is arguably worse than RocksDB's explicit stalls because failures show up as latency spikes rather than clean throttling.

Postgres's checkpoint pressure#

Postgres isn't an LSM but has an analogous problem: dirty pages have to be flushed before their WAL can be recycled. If the page cleaner falls behind:

  • max_wal_size gets hit, forcing an immediate checkpoint.
  • The immediate checkpoint tries to flush all dirty pages at once, saturating I/O.
  • Query latency spikes for the duration.

The fix is to tune checkpoint_timeout and checkpoint_completion_target so that regular checkpoints spread out the I/O.

Takeaway#

Under sustained overload, an LSM will always fall behind eventually — physics. The question is what happens then:

  • Clean back-pressure (RocksDB) — predictable, actionable.
  • Latency spikes and cascading GC (Cassandra) — hard to diagnose.
  • Checkpoint storms (Postgres) — traditional and easy to profile.

Pick the failure mode you can operate around.