Login and upload should work now we are monitoring

We forced restart our database and it came back, the thing we dont get yet is: does it will last as previous downtime seems not tight to any of our activity or our user. CPU and RAM where normal before the crash we are waiting for supabase answer

supabased reported the incident here https://status.supabase.com

Our Supabase database stopped to answer today, nothing changed on our side. updates are unaffected but upload and login are down

This issue was fully resolved

The issue is now resolved

We found the issue, the webhook queue had 55k row and our consumer tried to send all the event at once. this made the consume exaust the connection number of the DB we cleaned up the Queue is the event list where older than a month for most

We released an update in webhook that made our Primary Database unresponsive. Upload and console are disrupted. Download update is still up and working Set channel in old plugin < 7.20 are affected

Our DDOS portection may have put your app in cache with 429 status (default when our DB cannot answer) If this is not resolved for you, or you need urgent release, please upload a version again this should clear cache. If you are still stuck please reach out we can fix manually

The issue is fully resolved

We are now back to 10% CPU as expected and are monitoring

We identified an issue on our plugin endpoint. An app that do not pay capgo anymore had a spike of Usage and still have ur plugin installed. The spike of usage did spamed our DB to send emails to the customer to notify him about the issue. We Implemented a change to ensure we cache the query and prevent overload.

Since this morning our CPU skyrocketed to 100% usage it seems correlated to Supabase issue reported here https://status.supabase.com We are monitoring and looking for solutions with Supabase teams. Updates already in our replicate will work without issue but new upload or switch channel are impacted

All Replicates are now up to date and use the latest setting who should prevent the unsync to happen again. To ensure this will not happen again we:

• Doubled the size of the primary
• Made the subscribtion auto retry to reconnect faster
• Changed the way Channel work with latest version and prevent it bloat our DB
• Ensure we auto delete in DB completely bundle and delta update, to make the replicate lighter
• Made the script to recreate a replicate faster and more resilient

At 17:41 Hong kong was finally getting delta updates as expected

South america is now completely in sync, we are looking now to Hong kong for ready status on delta updates

All Replicate are correct replying for Zip updates, only delta update are out of sync in South america And in hong kong, we are copying the data right now

Europe, south America and India are back In sync, North america is on the watch, Oceanian and Hong kong are waiting

We have a current running issue since this morning. The read replicator of our database is not replicating properly, so some data is missing. That's why, right now, the channel is not properly responding.

hyperdrive resolved the situation

Since Cloudflare issue our monitoring is still showing issue on certain endpoint while we are up an functional. To not loose history we are changing URL to borken URL until mark as fully down and will revert when it's the case in hope to fix the monitoring

all issue are solves by cloudflare

Cloudflare is experiencing an internal service degradation. Some services may be intermittently impacted. We are focused on restoring service. We will update as we are able to remediate. More updates to follow shortly. https://www.cloudflarestatus.com

The issue is fully resolve worldwide. Our Device Database was the one cause trouble not our postgres Detailled Postmortem with detail will be published soon A lot of work was done to optimize Postgres and our cache. Our Infra multi region is now used in production. You will see now faster updates than ever

We are monitoring the changes and the latency worldwide

We doubled the DB size to ensure it can support all connections

We identied the problem, our Postgres database has too many open Connection and had to drop some to keep working, that why some user got issue but not all

We splitted and deployed in 6 region our backend this morning Since then some region started to timeout

The issue is fully resolved

We are currently monitoring and expeting all endpoint to be back to normal

We did fix the configuration.

Our user reported some Network failures

We found out our last deployment at 6:45PM today had a wrong configuration. This deployment included new Read replicate configuration. This was made to handle the load we currently have of 50M devices daily.

This new configuration had issue, this was concerniing only 30% of all request.

THe monitoring is now correct and report uptime properly

Currently monitoring it

We updated the hyperping config to send proper value this should solve our issue

We found out the issue was not the endpoint itself but the way we ping it. We send an invalid stats value in purpose to not save it in our DB. From this weekend invalid value are refused by our API.

We found out this weekend after latest release Hyperping started to report stats endpoint was offline.

The problem has been fully resolved

The Capgo team has deployed a new version of the backend and we will continue to monitor the issue

The Capgo team has manually changed the URL of "api.capgo.app" to the correct backend. This should resolve the issue for the moment. We are working on a long-term resolution.

The problem has been identified. When a commit was pushed to the development branch, the development/preprod build of Capgo was accidentally assigned to the production route of "api.capgo.app" causing this outage.

The Capgo team has been notified about a problem with apikeys that occurs on the CLI. We are actively investigating and trying to resolve it as soon as possible. Updates are working as expected

Long ago we set a pg_temp in "Extra search path" to debug with Supabase team our issue with bloated pg_temp schema. We never removed it and when we updated our instance who upgrade PostgREST to v13 this became a problem and crashed PostgREST in loop. This Change was made years ago and forgotten. We are sorry for this, and will be more careful to revert our test in prod when we have issue.

The issue is resolved, it seems a config got modified in supabse without us chaging it, we are looking in supabase to find a solution to never have this again

Supabase seems to have fix the issue ! We are waiting final update from them

Supabase upgraded plan got effective we got quick answer they are looking into it finally

We got answer in Supabase Discord but it just said wait for support to answer...

We upgraded Supabase plan to enterprise support level to try to get faster response.

We tried to reboot our Supabase infrastructure, enable disable API, nothing works it seems our only solution is to wait Support to answer

We tried to rotate ourself the JWT with what we have avaiable in Supabase UI and nothing works. We opened a ticket in they support, tagged them on X and DM the CEO.

We identified the issue. It's due to the change Supabase is rolling for their JWT auth system. We currently dont have any control on that and reached them to find a solution. You user can still receive updates, certain feature from CLI still works

After doing maintenance on our Database, update to a minor version. Login started to fail in the webapp. Updates are working as expected

We are fully operational

We managed to switch back to Supabase for updates, and latency is back to normal

We are stuck with D1 read replicate while Supabase service is restored, as Cloudflare has issue deploying. update are still deployed but slower than usual

Cloudflare started to have issue as well https://www.cloudflarestatus.com

We switched to our Read replicate in Cloudflare D1 to maintain updates.

Our database provider Supabase has currently network issue you might experience some delay in downloading updates. https://status.supabase.com

We are receiving reports of increased latency across a number of Capgo APIs. The team is currently looking into this now.

The issue is fully resolved

We are monitoring this changes

We improved our code and now take bigger advantage of Hyperdrive when directly connecting to our Supabase instance. For more Endpoint this makes D1 optional and lower the pressure in Supabase.

Our Cache system made in Cloudflare D1 started to degrade in quality, and made Cache query slower than direct DB connexion.

We started to see Issue in US region with timeout

We did unplug the Cache and only keept direct DB connexion

The upgrade went well

We upgrade our Supabase instance to latest to maintain security, updates are still distributed by our read replicate

This incident has been resolved.

Cloudflare has issue https://www.cloudflarestatus.com/incidents/25r9t0vz99rp

This impact currently our : Upload system Statistics SetChannel

Download update are not affected. we are monitoring actively All Cloudflare services have been restored and are now fully operational. We are moving the incident to Monitoring while we watch platform metrics to confirm sustained stability.

It latest 5 min all went well

We are updating the DB, upload will be disabled for 6 min, update will still be distributed by our read replicate

Issue resolved

Old update are distributed but none of the new one where visible for users. We did disable the D1 cache and all work again.

Our cache layer D1 stopped to receive updates as Cloudflare re actived a feature they disabled a month ago ( DB replication) without our consent.

Our cache layer stopped to respond