Delay on machine job startup

Incident Report for CircleCI

Resolved

Our logs have indicated that the startup of the jobs is now back to a normal amount of time.

We are still working with our cloud provider to monitor in case of a reoccurrence.

Thank you for your patience.

Posted Apr 19, 2022 - 17:11 UTC

Update

We will continue to monitor the situation and we are now able to provide 3 of 5 of the availability zones.

Thank you for your patience.

Posted Apr 19, 2022 - 16:58 UTC

Update

We have mitigated the effects of the issue with our cloud provider, our customers should see their jobs starting with delays of up to 30s.

We continue to work closely with our cloud provider and hope to see recovery soon.

We will continue to monitor the situation during the daily maximum utilization while we run with reduced availability zones (2 of 5).

Thank you for your patience.

Posted Apr 19, 2022 - 16:24 UTC

Monitoring

We have mitigated the effects of the issue with our cloud provider, customers should see their jobs starting with delays of up to 30s.

We are continuing to work closely with our cloud provider and hope to see recovery soon.

Thank you for your patience on this.

Posted Apr 19, 2022 - 15:51 UTC

Update

We have identified this issue as a problem with allocating encrypted storage.

We are making configuration changes to allocate instances in zones that do not have this issue.

Posted Apr 19, 2022 - 15:35 UTC

Update

We are continuing to work closely with our cloud provider to help resolve this issue.

We are sorry for the delays and are doing all we can to resolve this as soon as possible.

Posted Apr 19, 2022 - 15:17 UTC

Identified

We have identified an issue in our cloud provider which is causing us not to be able to allocate instances.

We have moved as much traffic as we can to alternate cloud providers.

Posted Apr 19, 2022 - 14:57 UTC

Update

We are continuing to investigate this issue.

Posted Apr 19, 2022 - 14:44 UTC

Investigating

There is a delay for some machine jobs and our team is currently looking into this.

This is due to us having trouble allocating instances in our cloud provider.

Posted Apr 19, 2022 - 14:25 UTC

This incident affected: Machine Jobs.