Incident 2023-12-04: Data leak and loss in some free tier databases

What Happened?

Approximately 0.07% of managed databases experienced misconfiguration with empty backup identifiers, causing both data exposure and potential data loss. The problematic system change occurred November 20th, with backups used to restore databases on December 1st. The issue was discovered and reported December 4th at 8:10 AM UTC and resolved by 9:17 AM UTC.

Root Cause

Free tier databases automatically scale to zero after one hour of inactivity, then restart upon receiving requests. When Fly.io lacked resources to restore processes, Turso would rebuild machines from S3 backups using unique identifiers.

A migration bug caused some databases to be assigned empty backup identifiers. Rather than pointing to s3://bucket/backup_id/, affected databases referenced s3://bucket//, creating a shared backup storage location.

When databases failed to scale back up, recreation from this shared null location caused both data loss and exposure. The remediation involved re-running migrations with correct parameters and restoring affected databases from December 1st backups—discarding any subsequent writes considered potentially shared.

Affected Database Identification

Engineers queried metadata for databases with invalid backup IDs and scanned object storage to confirm no other systems were compromised.

Remediation and Prevention

The team committed to implementing: additional control plane and data plane validation checks; stricter configuration monitoring with self-healing capabilities; improved deployment methods protecting backup identifiers; and enhanced security incident notification mechanisms.