Google's London report: three HA assumptions that failed

Most coverage of Google's incident report on the October 20 London outage stopped at the root cause, and it is a good one. A programmable logic controller took its undervoltage reading from the line side of a breaker that had been deliberately opened, so when the live feed died it still saw voltage on the dead path, decided the utility was healthy and never told the generators to start. The UPS batteries carried the load until their reserves ran out, and the data halls went dark. Google rates that bridge at roughly five minutes, which is how long it expects to need to get generators running.

That is Google's problem, and it says it has fixed it. The part worth a customer's time sits further down the same document, where Google is unusually direct about what broke in zones that never lost power. Each failure maps onto an assumption most high-availability designs make.

Capacity in the surviving zones

Google is clear that a VM is a zonal thing, and that building a path out of a dead zone is the customer's job, not its own. Then it records what happened when customers tried to use those paths: some attempts were, in its words, "unsuccessful due to capacity constraints", with the difficulty of replicating the underlying storage named alongside.

GKE tells the same story from the other end. Worker nodes did scale up in europe-west2-a and europe-west2-c, but Google says available capacity limited how far those scale-ups got, and regional API operations failed more often because of the reduced capacity and the extra load on the APIs underneath. One of the two primary causes Google names for impact outside the failed zone is instantaneous demand surging onto the neighbouring zones.

A failover plan is a claim on capacity you do not own and have not reserved. When a zone dies, everyone's plan calls that capacity at the same second.

Replica placement that was not what it said

The cleanest illustration comes from Google's own product. Apigee X returned elevated 5xx errors for 7 hours and 19 minutes, and the report is candid about why. The service is built to survive losing one zone, because its datastore is supposed to keep a replica in each of the region's three zones. A scheduling misconfiguration had put two of the three in the zone that lost power. The datastore lost quorum, and instead of failing over, the runtime failed requests.

Two of three in one zone means that zone holds the majority. The design was three zones; the reality was one. Google has since rebalanced the fleet, added monitoring to alert if replicas drift across zones, and says it will change Apigee X to enforce that all three replicas always run in three different zones.

If Google's own managed service drifted into that state without anyone noticing, an estate of self-managed quorum systems deserves the same audit.

Control planes that crossed the boundary

Google's second named cause of out-of-zone impact is control-plane dependencies that cross zonal boundaries. BigQuery lost roughly 5 hours and 9 minutes, largely because a critical data plane job depended on autoscaling that could not scale during the failover. Customers with nothing in the dead zone still saw about 1% error rates on Job API calls, which Google attributes to throttling applied too broadly. AlloyDB regional primaries waited up to 95 minutes before automated failover even started, because the notifications that trigger it were late, and roughly 4% stalled and needed manual recovery.

The contrast is Cloud Bigtable, which Google says behaved as expected. Multi-cluster instances using HA routing failed over cleanly, and the reason it gives is the one that matters: the replica clusters and the control plane were still up, with enough spare capacity to take the redirected load. Capacity and an independent control plane, in other words, are what made the failover real.

One more thing you were not told

The facility was not in its designed state when the cable failed. After a quarterly generator load test on October 14 found problems, feeder B was physically opened for repairs, leaving the site in what Google calls a temporary single-feed bridged maintenance posture. A 2N site was running 1N, and no customer knew.

Google says the main impact was resolved within 6 hours and 8 minutes, though its own per-service figures run longer, and that it found no impact outside europe-west2. It is expanding disaster-recovery testing to full cluster power-loss scenarios.

What IT teams should do

• Ask your cloud account team what happens to your failover if a zone's worth of demand arrives at once, and whether anything you run holds reserved capacity rather than best-effort.
• Audit replica placement for every quorum system you operate, and alert on drift rather than checking once. Two of three in one zone is a single point of failure wearing a three-zone label.
• Map which control-plane calls your recovery path depends on, and test the path with the control plane degraded, not just the data plane gone.
• Check that automated failover actually fires, and time it. A 95-minute notification delay is an outage even when the mechanism works.
• Treat a zone as capable of total loss, not graceful degradation, and rehearse a full-power-loss scenario rather than a clean zone drain.

Sources: Google Cloud incident report, The Register, The Stack.