Koyeb - Notice history

All systems operational

Control Panel - Operational

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2023
Oct 2023
Nov 2023

API - Operational

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2023
Oct 2023
Nov 2023

Build / Provisioning - Operational

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2023
Oct 2023
Nov 2023

Third Party: Github → API Requests - Operational

Third Party: Github → Webhooks - Operational

Third Party: Cloudflare → Cloudflare Sites and Services → API - Operational

Third Party: Docker → Registry - Operational

Third Party: Quay → Registry - Operational

Notice history

View current status

Nov 2023

Services down due to hypervisor failure
ResolvedPartial outage1 hour 9 minutes
  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC

    We implemented a fix and are currently monitoring the result.

  • IdentifiedUTC

    We are continuing to work on a fix for this incident, the situation is improving.

  • InvestigatingUTC

    Somes services deployed in FRA are unavailable due to a failure of the underlying hypervisors.

    Our engineering team is investigating.

Services down due to multiple hypervisor failure
ResolvedMajor outage1 hour 19 minutes
  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC

    We implemented a fix and are currently monitoring the result.

  • IdentifiedUTC

    We are continuing to work on a fix for this incident.

    Some services are still experiencing issues and are not re-scheduled yet.

  • InvestigatingUTC

    Multiple services services deployed in Frankfurt are unavailable due to a failure of the underlying hypervisors.

    Hypervisors failures happen and services should be automatically migrated - in this case they didn’t automatically recover.

    We are also seeing increase of 502 error due to some load balancer failure.

    Our engineering team is investigating.

Instabilities on deployment engine
ResolvedDegraded performance5 hours 15 minutes
  • ResolvedUTC

    This incident has been resolved.

  • InvestigatingUTC

    We are observing an increase of failing deployments and some delays in provisioning in Frankfurt. We are currently investigating the issue

Impossible to add new custom domains
ResolvedDegraded performance43 hours 50 minutes
  • ResolvedUTC

    This incident has been resolved.

  • IdentifiedUTC

    We are continuing to work on a fix for this incident.

  • InvestigatingUTC

    Adding new custom domains is currently unavailable. Existing domains are operational.

    The problem has been identified. However, it is in the hands of a third party service so we cannot commit on a resolution date yet.

New services stuck in Starting status
ResolvedDegraded performance2 hours 31 minutes
  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC

    We implemented a fix and are currently monitoring the result.

  • IdentifiedUTC

    New services on the platform are not reaching Healthy - they get stuck in Starting.

    The problem has been identified and a fix is currently being deployed.

Oct 2023

OCT17
Database update
CompletedMaintenance2 hours
  • CompletedOctober 17, 2023 at 10:00 AMUTC

    Maintenance has completed successfully

  • In progressOctober 17, 2023 at 8:00 AMUTC

    Maintenance is now in progress

  • PlannedOctober 17, 2023 at 8:00 AMUTC

    We are planning a database update at 8:00 UTC.

    We might experience an increased rate of API call failure, the control panel might be unresponsive for a few minutes. Provisioning may be delayed for a few minutes.

    No impact on running services.

OCT6
Runtime and build logs maintenance
CompletedMaintenance26 minutes
  • CompletedOctober 06, 2023 at 3:16 PMUTC

    Maintenance has completed successfully.

  • PlannedOctober 06, 2023 at 2:50 PMUTC

    We will operate a maintenance of our logs backends.

    As a result, recent logs will not be delivered for the duration of the maintenance. No data loss is expected: those logs will be accessible after the maintenance is completed.

Sep 2023

Critical host failure in FRA
ResolvedPartial outage4 hours 56 minutes
  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC

    We successfully recovered the impacted servers.

    Some workload are still being recovered automatically.

  • IdentifiedUTC

    We identified the root cause of the issue, some of the physical servers are lost because of high load, it seems that some reschedule of the impacted workload triggered a cascaded failure on other nodes.

    We are actively trying to recover access to the impacted servers.

    Some workload are still impacted and will soon recover.
    New build and container start are also impacted.

  • InvestigatingUTC

    We are seing a high increase of container restart and container boot failure in FRA.

    We are currently investigating this incident.

Previous

Sep 2023 to Nov 2023

Next