Skip to content

RocksDB tuning for tiny writes#

2024-02-14

Tuning RocksDB for a workload of ~100-byte keys and ~50-byte values with a high write rate. The default settings are optimized for larger values; tiny writes have some peculiarities.

What actually mattered#

The knobs that had the biggest impact for me, in rough order:

  1. write_buffer_size + min_write_buffer_number_to_merge — Larger memtables let you merge tiny writes before they get to L0. Default 64 MB was way too small; 256 MB with min_write_buffer_number_to_merge=2 cut L0→L1 compaction by about 40%.

  2. level0_file_num_compaction_trigger — Bumping from 4 to 8 helped a lot, because L0 files were tiny and there was no benefit compacting them individually. Compact them in bigger batches.

  3. target_file_size_base and max_bytes_for_level_base — Kept the default ratio (target_file_size_multiplier=1) but bumped base to 128 MB. Fewer, larger SST files at each level.

  4. compression_per_level — [kNoCompression, kNoCompression, kLZ4Compression, ...] — no point compressing L0/L1 that will compact away quickly, but LZ4 on the deeper levels was almost free.

What didn't help#

  • Bumping max_background_compactions past 4. My SSDs saturated at 4 threads.
  • Enabling direct_io. My kernel's page cache was doing a fine job.
  • enable_pipelined_write=true — no visible effect for this workload.

Reference numbers#

Before tuning:
  Sustained write throughput: 45k ops/sec
  L0 files: 12-15 typically
  Compaction stall %: 8-12%

After tuning:
  Sustained write throughput: 78k ops/sec
  L0 files: 2-4 typically
  Compaction stall %: <1%

Box was a 16-core Xeon E5-2660 v3 with two NVMe SSDs in software RAID-0.

Further reading#

  • The RocksDB tuning guide is the actual source of truth. Everything else is old.
  • Facebook's original RocksDB in Production blog series.