All systems operational

Control Panel
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
API
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

Build / Provisionning

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Third Party: Github → API Requests
Operational
Third Party: Github → Webhooks
Operational
Third Party: Cloudflare → Cloudflare Sites and Services → API
Operational

Third Party: Docker → Registry

Operational
Third Party: Quay → Registry
Operational

Notice history

Nov 2023

Services down due to hypervisor failure
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    Identified

    We are continuing to work on a fix for this incident, the situation is improving.

  • Investigating
    Investigating

    Somes services deployed in FRA are unavailable due to a failure of the underlying hypervisors.

    Our engineering team is investigating.

Services down due to multiple hypervisor failure
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

    Some services are still experiencing issues and are not re-scheduled yet.

  • Investigating
    Investigating

    Multiple services services deployed in Frankfurt are unavailable due to a failure of the underlying hypervisors.

    Hypervisors failures happen and services should be automatically migrated - in this case they didn’t automatically recover.

    We are also seeing increase of 502 error due to some load balancer failure.

    Our engineering team is investigating.

Instabilities on deployment engine
  • Resolved
    Resolved

    This incident has been resolved.

  • Investigating
    Investigating

    We are observing an increase of failing deployments and some delays in provisioning in Frankfurt. We are currently investigating the issue

Impossible to add new custom domains
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

  • Investigating
    Investigating

    Adding new custom domains is currently unavailable. Existing domains are operational.

    The problem has been identified. However, it is in the hands of a third party service so we cannot commit on a resolution date yet.

New services stuck in Starting status
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    Identified

    New services on the platform are not reaching Healthy - they get stuck in Starting.

    The problem has been identified and a fix is currently being deployed.

Oct 2023

Database update
  • Completed
    October 17, 2023 at 10:00 AM
    Completed
    October 17, 2023 at 10:00 AM

    Maintenance has completed successfully

  • In progress
    October 17, 2023 at 8:00 AM
    In progress
    October 17, 2023 at 8:00 AM

    Maintenance is now in progress

  • Planned
    October 17, 2023 at 8:00 AM
    Planned
    October 17, 2023 at 8:00 AM

    We are planning a database update at 8:00 UTC.

    We might experience an increased rate of API call failure, the control panel might be unresponsive for a few minutes. Provisioning may be delayed for a few minutes.

    No impact on running services.

Runtime and build logs maintenance
  • Completed
    October 06, 2023 at 3:16 PM
    Completed
    October 06, 2023 at 3:16 PM

    Maintenance has completed successfully.

  • Planned
    October 06, 2023 at 2:50 PM
    Planned
    October 06, 2023 at 2:50 PM

    We will operate a maintenance of our logs backends.

    As a result, recent logs will not be delivered for the duration of the maintenance. No data loss is expected: those logs will be accessible after the maintenance is completed.

Sep 2023

Critical host failure in FRA
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We successfully recovered the impacted servers.

    Some workload are still being recovered automatically.

  • Identified
    Identified

    We identified the root cause of the issue, some of the physical servers are lost because of high load, it seems that some reschedule of the impacted workload triggered a cascaded failure on other nodes.

    We are actively trying to recover access to the impacted servers.

    Some workload are still impacted and will soon recover.
    New build and container start are also impacted.

  • Investigating
    Investigating

    We are seing a high increase of container restart and container boot failure in FRA.

    We are currently investigating this incident.

Sep 2023 to Nov 2023