Skip to content

Cassandra tombstone reclamation#

2024-01-08

Cassandra deletes create tombstones — special markers with a TTL — rather than removing data immediately. Tombstone accumulation is one of the most common Cassandra ops problems.

Why tombstones exist#

Cassandra replicates data across nodes. If a node is down when a delete happens, then comes back up while still holding the old data, that node would resurrect deleted rows unless the delete was recorded somewhere.

Tombstones solve this: the delete is recorded as a tombstone with the delete timestamp. During anti-entropy repair, tombstones propagate; nodes with the old data see the newer tombstone and delete their local copy.

The gc_grace_seconds window#

Tombstones can't be reclaimed until gc_grace_seconds has passed since the delete. Default: 10 days. Why?

Because repair (the process that syncs tombstones between nodes) is expected to run within that window. If you reclaim a tombstone before all replicas have seen it, a stale replica can resurrect the data.

When it goes wrong#

  • Skipped repairs: if you don't run nodetool repair weekly, tombstones build up.
  • Range deletes (DELETE FROM t WHERE k > x AND k < y): create a single logical tombstone that covers the whole range. Reads that traverse this range have to skip everything.
  • Column-level tombstones in dense partitions: SELECT operations can read thousands of tombstones before finding live data.

Cassandra 3.0+ tracks tombstone counts per read and warns/errors at 1000/100000 by default (tombstone_warn_threshold, tombstone_failure_threshold).

Practical mitigations#

  • Model to avoid deletes: use TTL on inserts if the data expires naturally.
  • Time-window compaction strategy (TWCS): for time-series data, don't intermingle old and new data in the same SSTable.
  • Aggressive repair: use Cassandra Reaper or similar to guarantee weekly repair.
  • Shorter gc_grace_seconds: only if you trust repair to complete within the window.

Reference#

  • Cassandra docs on tombstones.
  • DataStax blog posts on "Deletes and Tombstones."