Issues with S3 and Job Starts
Incident Report for CircleCI

AWS S3 was offline from around 9:30AM PST until roughly 2PM PST.

The failure of S3 caused further disruptions to AWS EBS, EC2 and ECR as S3 stores volumes, container images and AMIs. This disruption made CircleCI unable to start new servers, or new build containers. We were not an isolated incident, our upstream resources were also severely impacted. As one example, we stopped receiving any web hooks from GitHub due to the disruption.

The immediate impact of the outage was an increase in the backlog of new jobs. As this happened during the time of the day we typically ramp to meet demand, our server count was under capacity which slowed our ability to process jobs in the queue.

While we waited for AWS to restore S3, we worked to ensure we did not lose any current servers and prepared for the influx of build jobs once web hooks were restored. Our processing of the backlog of jobs continued for approximately two hours after AWS restored services as we worked within the AWS enforced recovery limits to ramp up our server capacity.

AWS S3 Service disruption report: https://aws.amazon.com/message/41926/

Posted 8 months ago. Mar 06, 2017 - 09:44 PST

Resolved
We have seen no remaining issues and as such we are marking this incident as resolved. Thank you for your patience and continued support.
Posted 8 months ago. Feb 28, 2017 - 17:04 PST
Update
The backlog has been processed and we are going to monitor the situation for another 20 minutes to ensure that we are in the clear.
Posted 8 months ago. Feb 28, 2017 - 16:44 PST
Update
The backlog of queued builds continues as we maintain a higher level than normal of resources. We will update again in 30 minutes.
Posted 8 months ago. Feb 28, 2017 - 16:28 PST
Update
The backlog of queued builds is being processed. We've brought additional resources online to meet demand and will continue to monitor. We anticipate service to return to normal soon. Next (and hopefully last) update in 30 minutes.
Posted 8 months ago. Feb 28, 2017 - 15:50 PST
Monitoring
AWS S3 is operating normally again. We will continue to bring additional resources online to process the backlog of builds and continue to monitor the situation closely. We'll update again in 30 minutes.
Posted 8 months ago. Feb 28, 2017 - 15:16 PST
Update
We are continuing to work on the backlog of builds while monitoring the AWS S3 status. We will update again in 30 minutes.
Posted 8 months ago. Feb 28, 2017 - 14:15 PST
Update
We're continuing to see improvement, however our systems are still impacted as a result of the S3 issues. Next update in 30 minutes. Thanks for your patience as we continue to monitor the situation.
Posted 8 months ago. Feb 28, 2017 - 13:42 PST
Update
We are starting to see signs of improvement. AWS expects to see lower error rates within the hour. Will update again in 30 minutes.
Posted 8 months ago. Feb 28, 2017 - 13:02 PST
Update
AWS believes they have identified the cause of the S3 issue and are working hard on implementing a fix. We'll update again in 30 minutes.
Posted 8 months ago. Feb 28, 2017 - 12:13 PST
Update
The AWS S3 availability issue persists. We'll continue to monitor and update again in 30 minutes.
Posted 8 months ago. Feb 28, 2017 - 11:31 PST
Update
We're continuing to experience issues with AWS S3 and are monitoring the situation closely. We'll update again in 20 minutes. Thank you for your patience.
Posted 8 months ago. Feb 28, 2017 - 11:14 PST
Update
The issue with AWS S3 is still ongoing, we are working on keeping our fleet ready to respond when the event is over
Posted 8 months ago. Feb 28, 2017 - 10:39 PST
Identified
We have identified the issue with our upstream providers and are monitoring the situation
Posted 8 months ago. Feb 28, 2017 - 10:19 PST
Update
We are seeing widespread issues with AWS and GitHub which are impacting our ability to handle builds
Posted 8 months ago. Feb 28, 2017 - 09:53 PST
Investigating
We are currently investigating this issue.
Posted 8 months ago. Feb 28, 2017 - 09:51 PST