Getting a request to a live backend is three decisions, not one
A user types your domain and presses enter. Before a single byte of their request reaches a machine that can answer it, three independent routing systems have a say — and each one operates on a different clock. Anycast decides at the network layer, in seconds to minutes, on BGP's schedule. DNS failover decides at resolution time, on TTLs you publish but do not enforce. Proxy health checks decide per connection, in single-digit seconds, inside your own load balancer. They fail in completely different ways, and the failures stack. Understand one layer and not the others, and you will eventually ship an outage that "should have failed over" and just sat there.
Layer 1: Anycast — same IP, many locations
With anycast, many physically separate servers announce the same IP prefix to the internet over BGP, each from a different location. A client opens a connection to one address; every router along the way independently picks the topologically nearest announcement and forwards the packet there. Nobody makes a per-request decision — the routing table itself encodes "go to the closest site that's still announcing." This is how 1.1.1.1 and most large CDNs work: one address, dozens of points of presence.
Failover is implicit. When a site goes dark, its router stops announcing the prefix (or an upstream withdraws the route), BGP reconverges, and traffic shifts to the next-nearest site still in the game. The catch is timing. Convergence is not instant — withdrawing and repropagating a route across the internet can take a few seconds to a couple of minutes, depending on peering, route-flap dampening, and how the withdrawal was triggered. And anycast only reacts when the route disappears. A server that is still up, still announcing its prefix, but returning 500s to every request will happily keep collecting traffic. Anycast detects dead networks. It is blind to dead applications.
It is also operationally heavy. You need your own IP space, an autonomous system number, and BGP sessions with multiple transit providers. Most teams get anycast indirectly — by sitting behind a CDN or a managed DNS provider that runs it — rather than operating it themselves.
Layer 2: DNS failover — and the TTL trap
DNS failover is the layer most teams configure by hand. A managed DNS provider health-checks your endpoints, and when the primary fails it starts serving a different A record pointing at the backup. Simple in principle. The problem: you do not control how long your old answer lives in caches.
A published TTL is a floor, not a ceiling. It is the minimum time a resolver is allowed to cache your record — nothing forces expiry on schedule, and some resolvers enforce their own minimum and refuse to cache anything below ~30 seconds even if you set a TTL of 5. Between your authoritative server and your user sits a stack of caches you cannot see: the ISP's recursive resolver, a corporate resolver, a container DNS sidecar, the home router, and the OS-level cache (systemd-resolved, the Windows DNS Client service, mDNSResponder on macOS). Any one of them can hold a stale answer past your TTL. So you publish 60 seconds, the primary dies, and a real slice of users keep resolving to the corpse for minutes.
There is a second trap that catches people who did everything right at the DNS layer: connection reuse. DNS is only consulted when a new connection opens. If your clients use HTTP keep-alive — most do — an existing socket to the dead backend keeps getting reused. DNS can update instantly and the request still lands on the dead host. No DNS setting fixes this. The fix is application-level retry logic that re-resolves and reconnects on failure.
The honest version: DNS failover is real and useful, but treat the TTL as a hint, never as a promise about when users will actually move. A passing health check on your secondary tells you the secondary is healthy. It tells you nothing about whether traffic has reached it.
Layer 3: Proxy health checks — the fast, precise layer
This is where failover is genuinely fast and genuinely correct, because it happens inside infrastructure where you control every parameter. A reverse proxy or load balancer — HAProxy, nginx, Caddy, Envoy — sits in front of a pool of backends, decides per connection which one to use, and yanks a sick backend out of rotation on its own.
Active health checks probe each backend on an interval whether or not real traffic is flowing. HAProxy's default check interval is 2 seconds; a backend is marked down after fall consecutive failures (default 3) and restored after rise successes (default 2). A hard backend failure is therefore detected and routed around in roughly 6 seconds — no DNS dependency, no cache to wait out, no BGP to reconverge. Passive health checks instead watch live traffic for errors. Cheaper, but they only notice a problem after a real request has already hit the bad backend.
The limit of this layer is reach. A load balancer can only fail over among backends it can see. If the load balancer's own IP is dead, or the entire site it lives in is gone, no amount of health checking saves you — that is the exact gap Layers 1 and 2 exist to cover. It is also why naive round-robin DNS — publishing several A records and letting clients rotate through them — is not real load balancing. Plain round-robin has zero health awareness; it hands out dead IPs as cheerfully as live ones, because DNS has no idea a server is down. Weighted records let you bias the split, but the caching problem from Layer 2 still rides along.
Where each layer belongs
Stack them by what they are actually good at. Anycast gets users to the nearest region and survives the loss of a whole site or network path — coarse, slow, global. DNS failover steers between regions or providers when an entire entry point dies, on the understanding that the cutover takes minutes, not seconds, for some users. Proxy health checks do the fast, precise, per-connection routing within a region; that is your primary defense against a single backend dying.
Most production setups run at least two of the three. A common shape: a network load balancer on one advertised IP per region, fronting proxies that do health-aware routing to the application servers behind them, with DNS failover as the cross-region escape hatch.
That two-layer shape is roughly how Beebeeb's own infrastructure is built. Our backend lives in Falkenstein, Germany. A network load balancer presents a single entry IP and forwards to two front-end nodes; each runs Caddy for TLS termination and reverse-proxies to local services, and an internal HAProxy layer does health-aware routing to the current database and cache primary — so a Sentinel-driven failover is followed within a couple of seconds by the proxy detecting the new master. We deliberately do not run anycast today. We are a single-region operator, and pretending otherwise would be marketing, not engineering. Wider geographic distribution is the kind of thing we'd put on our public roadmap rather than quietly imply.
None of this touches your data's confidentiality, which is the point. Traffic routing is an availability problem; encryption is a confidentiality one; they are solved at different layers and should not be confused. Because Beebeeb is end-to-end encrypted and zero-knowledge, the load balancer in front of our servers only ever shuffles ciphertext — it could route you to any node and still never see a plaintext byte.
The one rule that survives all three layers
Health checks tell you whether an endpoint is up. They say nothing about whether traffic has moved. Anycast can still be reconverging, your TTL can be honored to the second, your proxy can have already pulled the bad backend — and a stubborn keep-alive socket or a resolver clinging to a stale answer can keep pinning a user to a dead server. Test failover by watching where requests actually land, not by confirming the backup reports itself healthy. Those are two different facts, and the gap between them is where the 3 a.m. pages live.