Issues loading workflows UI

Incident Report for CircleCI

Postmortem

Yesterday we experienced an issue that caused workflows to be degraded for some customers for about an hour starting at 20:19 UTC, with service fully restored by 21:00 UTC.

The event was triggered by a period of heightened packet loss and latency at an upstream provider. We began to notice increased API error rates within our system, which caused workflow jobs to not start. Once the upstream provider returned to normal activity, workflows began processing and we were able to clear the backlog, while continuing to closely monitor.

Posted Aug 09, 2018 - 22:03 UTC

Resolved

All services are stable. We would like to thank you for your patience. Incident resolved.

Posted Aug 07, 2018 - 21:17 UTC

Update

Jobs are being processed normally. We will continue to monitor our systems for stability.

Posted Aug 07, 2018 - 20:51 UTC

Monitoring

We're experiencing a network failure with an upstream provider and are monitoring the situation.

Posted Aug 07, 2018 - 20:36 UTC

Update

We are seeing increased API error rates on our front-end. Some jobs may experience queueing.

Posted Aug 07, 2018 - 20:26 UTC

Investigating

We are investigating an issue with the workflows UI.

Posted Aug 07, 2018 - 20:19 UTC

This incident affected: CircleCI Dependencies (AWS) and Docker Jobs, Pipelines & Workflows, CircleCI UI.