August 26: WireGuard gateways broke after schema migration mismatch

Incident start: 18:10UTC

We ran a routine deploy of wggwd, the service that sets up customer WireGuard peers, which shouldn’t have impacted anything as the latest code was already deployed (or so we thought). Soon after the deploy we noticed new peer additions failing with a SQL error (wggwd runs a sqlite database on each server to cache peers).

This was tracked down to a new feature whose SQL migration had been merged but not deployed; wggwd was erroring on a select * that picked up more columns than the code expected. However, we were confused by the error, since it was not possible for the wggwd service to both have restarted to run migrations, and not restarted as to fail on the new schema. It turns out this was caused by our alerting, that runs a wggwd check command on an interval; the check command was incorrectly set up to run migrations when opening the database connection.

We closed out this incident by restarting wggwd everywhere to pick up the new code, fixing the check command so it doesn’t run migrations, and ensuring deployments actually restart the server process.