Increased wait times for machine executor jobs

Incident Report for CircleCI

Resolved

Machine executor jobs have resumed normal operation
Posted Aug 19, 2019 - 15:34 UTC

Monitoring

A fix has been deployed and machine executor allocation times should be back to normal
Posted Aug 19, 2019 - 15:20 UTC

Investigating

We are seeing another increase in apt errors which is now causing all VMs to fail to boot
Posted Aug 19, 2019 - 14:35 UTC

Monitoring

We have scaled up to deal with the rate of errors
Posted Aug 19, 2019 - 13:56 UTC

Identified

The issue has been identified and a fix is being implemented.
Posted Aug 19, 2019 - 13:29 UTC

Investigating

Due to an increase in failures connecting to the apt repositories, we are seeing increased wait times for Machine builds
Posted Aug 19, 2019 - 13:28 UTC
This incident affected: Machine Jobs.