Job Processing Delayed

Incident Report for CircleCI

Resolved

All executor job queues have returned to normal.

Posted Feb 12, 2020 - 17:06 UTC

Update

macOS customers may still experience some delays due to contention.

Posted Feb 12, 2020 - 17:00 UTC

Monitoring

Job queues have returned to their usual levels, customers with limited concurrency may still experience some delay.

Posted Feb 12, 2020 - 16:48 UTC

Update

Queue times are close to returning to normal levels and we are continuing to work on reducing them.

Posted Feb 12, 2020 - 16:34 UTC

Update

We are continuing efforts to reduce job queuing.

Posted Feb 12, 2020 - 16:14 UTC

Identified

We have resolved the underlying issue with our datastore, and job processing should be returning to normal.

Posted Feb 12, 2020 - 15:51 UTC

Update

We are continuing to investigate an issue with our underlying datastore. Delays may be up to 9 minutes per job.

Posted Feb 12, 2020 - 15:48 UTC

Investigating

We are currently investigating the cause of delayed jobs.

Posted Feb 12, 2020 - 15:35 UTC

Identified

Jobs are taking longer than expected to process.

Posted Feb 12, 2020 - 15:15 UTC

This incident affected: Docker Jobs, Machine Jobs, macOS Jobs, and Workflows.