Skip to content

TCP fast open in the datacenter#

2024-07-08

TCP Fast Open (TFO, RFC 7413) lets a client send application data in the SYN packet, saving one RTT on connection setup. Useful in a few narrow scenarios.

How it works#

  • First connection to a server: client and server negotiate a TFO cookie via SYN + SYN-ACK exchange. Standard 3-way handshake.
  • Subsequent connections: client sends SYN + cookie + application data in one packet. Server validates cookie, delivers data to application immediately.

Savings: 1 RTT on connection setup. For short-lived requests over WAN, this is significant (~50% latency reduction for a 1-request-1-response transaction).

Where it doesn't work#

  • First connection to any server: no savings. Need repeat visits.
  • Middleboxes: many NAT/firewall boxes strip TFO options. Deployment ratio in the wild is estimated at 30-40% for the option surviving to the server.
  • DNS not integrated: you still pay 1 RTT for DNS resolution unless you have a warm cache.
  • TLS 1.2 negates savings: TLS 1.2 requires 2 more RTTs anyway. TLS 1.3 (or 0-RTT) is where TFO stacks with other savings.

In the datacenter#

Actually mostly irrelevant. Datacenter RTTs are 100 µs; saving 1 RTT saves 100 µs. Not enough to justify the complexity.

Where it does help internally: cross-region service-to-service RPC where RTTs are 20-100 ms and connections are short-lived. gRPC (which uses HTTP/2 persistent connections) already amortizes handshake cost, so TFO's window here is narrow too.

Enable / disable#

# Linux: check status
sysctl net.ipv4.tcp_fastopen
# 1 = clients only, 2 = servers only, 3 = both

# Enable both
sysctl -w net.ipv4.tcp_fastopen=3

Applications also need to opt in via TCP_FASTOPEN_CONNECT (client) or TCP_FASTOPEN (server) socket options.

Reference#

  • Cheng et al., TCP Fast Open (RFC 7413).
  • Radhakrishnan et al., TCP Fast Open (CoNEXT 2011) — the original Google paper.