The API has been stable since the fix was deployed at 09:41 UTC. Between 03:15 and 09:41 UTC, some API requests failed with 503 database_unavailable, which blocked some CLI and CI bundle uploads. Database replication and update delivery to devices were not affected. A follow-up change that stops read requests from holding database connections is in review. No action is needed on your side. If a deploy failed during this window, please run it again.
Release 22.0.0 with the fix was deployed at 09:41 UTC. Database connections are being released normally again and API errors have stopped. We are monitoring.
Errors are continuing. We confirmed that all of the API's database connections are being held open by requests that only read data. We are deploying the merged fix now and preparing a broader one.
We found that a billing change released overnight keeps database connections open while it waits on external calls, which can use up the API's database connection pool. A fix has been merged and will ship in the next release.
Some API requests are taking 15–25 seconds and then failing with 503 database_unavailable. CLI and CI bundle uploads may fail. We are investigating.