London BGP outage

October 8 (05:41UTC)

In the process of upgrading our edge servers in LHR, it turned out we decommissioned the old hosts before confirming the new ones were correctly advertising routes over BGP (hint: they were not). The result was a loss of inbound connectivity through LHR edges for roughly 15 minutes. We restored service by bringing the old edge servers back online, and BGP re-established normally once they were up. Our internal health check for BGP state should have caught this before the old servers were shut down, but it was passing on the new hosts even though BGP wasn’t actually working. We’re investigating why that happened and tightening that check so this doesn’t slip through the same crack later on.