Skip to content

CockroachDB range management#

2024-05-28

CockroachDB (CRDB) shards its data into ranges — contiguous keyspace slices, replicated via Raft. Range management is where a lot of the interesting engineering happens.

Range basics#

  • Default range size: 512 MiB.
  • Each range is a Raft group with 3-7 replicas (usually 3).
  • Ranges are indexed in a two-level meta-range structure (meta1 → meta2 → data ranges).

Splitting#

When a range exceeds its target size, it splits. The split point is chosen to balance the two halves roughly evenly by size (not by row count, because row sizes vary). Actual mechanism:

  1. Compute the split key.
  2. Propose a SplitTrigger command through Raft.
  3. The command atomically:
  4. Creates two new range descriptors.
  5. Divides the local RocksDB keys between them (using RocksDB range deletions to trim, plus renaming ranges).
  6. Notifies the meta ranges.

Splits are relatively cheap because ranges share the same RocksDB instance — splitting just changes metadata about which keys belong to which range.

Merging#

The reverse: two adjacent ranges combine into one. Triggered when both are below the merge threshold (default: quarter of split size). More complex than splits because the two ranges may have different Raft groups; merging requires transferring leadership and moving replicas.

Rebalancing#

CRDB continuously rebalances ranges across nodes based on:

  • Store capacity utilization.
  • Range count per store.
  • Localities (rack awareness).
  • Follower reads / lease placement.

A background rebalance queue on each node scans local ranges and proposes replica movements. Movement uses a change replica operation: add new replica, remove old, all via Raft configuration changes (joint consensus).

Practical wisdom#

  • Range sizes vary in practice — a table with a hot prefix might have 10 GB ranges before splits catch up.
  • Lease placement matters a lot for latency; check the crdb_internal.ranges view to see where leases live.
  • Cross-region latency: choose "regional survivability" if you can, not "region survival." The latter puts you at 3× WAN latency for writes.

Reference#

  • Taft et al., CockroachDB: The Resilient Geo-Distributed SQL Database (SIGMOD 2020).
  • The CRDB source at pkg/kv/kvserver/.