Elevated rate of 404 and 502 on healthy services

Resolved

Partial outage

Started about 2 years agoMay 29, 2024Lasted about 2 hoursMay 29, 202410:28 AM12:20 PMUTC

Affected

North America

Washington, D.C. - WAS

San Francisco - SFO

Europe

Frankfurt - FRA

Paris - PAR

Updates

Resolved
May 29, 2024 at 12:20 PMUTC
Resolved
May 29, 2024 at 12:20 PMUTC
The incident is now resolved, we are sorry about the impact, and we are available to discuss the details if needed.
Monitoring
May 29, 2024 at 12:18 PMUTC
Monitoring
May 29, 2024 at 12:18 PMUTC
We implemented a fix and have re-scheduled most of the impacted failing service.

The initial incident triggering 404 was handled very shortly after it was detected, we noticed that some users were still experiencing some 503 due to an automatic garbage collection of some network resources.

Then we implemented a fix to recover the data that were garbage collected.
Identified
May 29, 2024 at 11:22 AMUTC
Identified
May 29, 2024 at 11:22 AMUTC
The recovery was done under 5m for most app, but a portion of user were still facing 503, we are currently trying to resolve this issue.
Investigating
May 29, 2024 at 10:28 AMUTC
Investigating
May 29, 2024 at 10:28 AMUTC
We are currently investigating this incident, it seems it was caused by a configuration error from our side.

Koyeb - Elevated rate of 404 and 502 on healthy services – Incident details