Skip to content

FoundationDB architecture notes#

2024-10-11

FoundationDB (FDB) has an unusual layered architecture that's worth understanding even if you don't use it — it inspired a lot of subsequent systems.

The layers#

FDB is deliberately narrow at its core:

  • Core: ordered key-value store with ACID transactions.
  • Layers: everything else (SQL, documents, tuples, blob store) built on top.

The core doesn't know about SQL, tables, or types. It's just: set(key, value), get(key), clear_range(begin, end), and multi-key transactions.

The processes#

An FDB cluster is composed of several process roles:

  • Coordinators (3-9): store cluster configuration. Paxos-like.
  • Cluster controller: elected leader that assigns roles.
  • Master: manages the transaction lifecycle, hands out commit versions.
  • Proxies: handle client requests, serialize commits.
  • Resolvers: check for read-write conflicts between concurrent transactions.
  • TLogs: durable write-ahead log for the whole cluster.
  • Storage servers: actual key-value storage.

Note: coordinators are separate from the master. The master role is a single-writer that can move.

Transactions#

FDB uses optimistic MVCC with a global read version:

  1. Client gets a read version from the proxy (some recent commit version).
  2. Client reads at that version, tracking read set.
  3. Client sends read set + write set to proxy.
  4. Proxy assigns a commit version, sends conflict check to resolver.
  5. Resolver checks if any concurrent commit overlaps the read set.
  6. If no conflict: proxy writes to TLogs, TLogs push to storage servers.

Commit latency is dominated by the TLog fsync, typically 1-2 ms in production.

The cool bit: deterministic simulation#

FDB is famous for its simulation testing framework: the entire cluster is compiled into a single-threaded event-driven simulator. You can replay a year of network partitions, disk failures, and clock skews in an hour. Bugs surface deterministically.

This is the reason FDB is famously bulletproof. Everyone else's distributed systems have random flaky tests; FDB has "let's simulate 10,000 hours of chaos overnight."

Reference#

  • Zhou et al., FoundationDB: A Distributed Unbundled Transactional Key Value Store (SIGMOD 2021).
  • The FDB documentation, in particular the architecture page.