Our APIs and control panel are currently unavailable due to a simultaneous failure of several hosts of our control cluster.

15% of services deployed in Frankfurt are unavailable due to a network failure of the underlying hypervisors. Hypervisor failures happen and services should be automatically migrated - in this case, they didn’t automatically recover. Our engineering team is investigating.

We identified a partial issue that impacted domain name resolution for services deployed in Frankfurt. Resolutions to several domains including mongodb.net subdomains would fail. This incident has been mitigated at 6:40 PM UTC by switching to an alternative resolver. We continue to monitor the results.

We just experienced an outage impacting the default provided domain name under *.koyeb.app. During a few hours, the query to this domain ended up having tls validation issue. This seemed to be caused by an issue with the automatic renewal of the certificate, we are doing some more investigation to find the root cause and prevent this from happening again. The custom domains were not impacted by this incident.