LSM write amplification revisited#

2024-11-03

The classical result is that a k-way merge over L levels with size ratio T gives you a write amplification of roughly L * T in the worst case. RocksDB's default of T=10 and 7 levels comes out to something like 70x on paper, but the number you actually observe is usually a lot lower because upper levels stay warm and compact more often than the theory admits.

I benchmarked a workload of 200-byte keys with a 90/10 read/write ratio on a bounded working set (about 40 GB) and got sustained WA of ~18x, not 70x. The culprit for the discrepancy is that the last two levels almost never compact for this workload — the data set is small enough that L5/L6 stay essentially stable, and each key only gets touched by a handful of L0→L1→L2 compactions.

rocksdb.total-sst-files-size    41 GB
rocksdb.compact-write-bytes    738 GB
observed WA                    ~18x
predicted WA (formula)         ~70x

Takeaway#

If you're capacity-planning for a bounded working set, don't blindly use the worst-case formula — measure it. If you're planning for a growing set that will eventually fill all levels, the formula is a fair upper bound.

The WiscKey paper is a nice read alongside this: separating values from the LSM tree gets you sub-5x WA at the cost of a range-scan penalty.

LSM write amplification revisited#

Takeaway#

Further reading#