All systems operational

Control Panel
100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%
Nov 2023100.0% uptime
Dec 2023100.0% uptime
Jan 2024100.0% uptime
API
100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%
Nov 2023100.0% uptime
Dec 2023100.0% uptime
Jan 2024100.0% uptime

Build / Provisionning

99.90% uptime
Nov 2023 · 100.0%Dec · 99.71%Jan 2024 · 100.0%
Nov 2023100.0% uptime
Dec 202399.71% uptime
Jan 2024100.0% uptime
Third Party: Github → API Requests
Operational
Third Party: Github → Webhooks
Operational
Third Party: Cloudflare → Cloudflare Sites and Services → API
Operational

Third Party: Docker → Registry

Operational
Third Party: Quay → Registry
Operational

Notice history

Jan 2024

Control panel unavailable
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

Delay in provisionning
  • Resolved
    Resolved

    This incident has been resolved.

    We had to perform an unplanned maintenance (involving a restart) on some LB components.

    Some services may have experience 502 errors during the maintenance.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    Identified

    The root cause has been identified.

    We are continuing to work on a fix for this incident.

  • Investigating
    Investigating

    We noticed a delay when provisioning or re deploying services.
    We are currently investigating this incident.

Dec 2023

Some service are no longer reachable
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We believe we found the root cause of the issue, more investigation is required, it seems to be related to one of the load-balancing components misbehaving under a certain load.

  • Investigating
    Investigating

    There is currently an elevated rate of 502 errors when requesting some services, we are currently investigating the root cause of the issue.

Provisioning delay
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

    The cluster is now back online and the builds should be able to restart.

    We tried to re-route the builds to another region during the outage, but we were sadly not able.

    We are still trying to understand the root cause of the issue.
    And we will try to understand why the re-routing of the build was not possible during the outage.

  • Identified
    Identified

    We have identified some provisioning delay for builds and deployments in the Frankfurt region.

    Our engineering team is investigating.

Nov 2023

Services down due to hypervisor failure
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    Identified

    We are continuing to work on a fix for this incident, the situation is improving.

  • Investigating
    Investigating

    Somes services deployed in FRA are unavailable due to a failure of the underlying hypervisors.

    Our engineering team is investigating.

Services down due to multiple hypervisor failure
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

    Some services are still experiencing issues and are not re-scheduled yet.

  • Investigating
    Investigating

    Multiple services services deployed in Frankfurt are unavailable due to a failure of the underlying hypervisors.

    Hypervisors failures happen and services should be automatically migrated - in this case they didn’t automatically recover.

    We are also seeing increase of 502 error due to some load balancer failure.

    Our engineering team is investigating.

Instabilities on deployment engine
  • Resolved
    Resolved

    This incident has been resolved.

  • Investigating
    Investigating

    We are observing an increase of failing deployments and some delays in provisioning in Frankfurt. We are currently investigating the issue

Impossible to add new custom domains
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

  • Investigating
    Investigating

    Adding new custom domains is currently unavailable. Existing domains are operational.

    The problem has been identified. However, it is in the hands of a third party service so we cannot commit on a resolution date yet.

New services stuck in Starting status
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    Identified

    New services on the platform are not reaching Healthy - they get stuck in Starting.

    The problem has been identified and a fix is currently being deployed.

Nov 2023 to Jan 2024