API and non-CircleCI account builds broken

Incident Report for CircleCI

Postmortem

The evening of the 19th of January a new version of an analytics library was merged into our code base and deployed around 9PM UTC.

Our customers started to notice builds not working around midnight UTC for any customer who was triggering them via our API, soon after we declared an incident. We determined the source of the issue we rolled back to a previous version.

We are always working to improve our system and tools and take seriously any impact the changes have to our customers. We appreciate the patience of the customers that were impacted by this change.

Posted Jan 31, 2017 - 19:00 UTC

Resolved

This incident has been resolved.

Posted Jan 20, 2017 - 01:38 UTC

Monitoring

We've resolved the issue with broken builds from API and non-CircleCI accounts, and we'll continue to monitor closely. If you have any builds marked as Not Running, hitting the rebuild button will fix the issue. If you see anything amiss, please reach out to support@circleci.com.

Posted Jan 20, 2017 - 01:13 UTC

Investigating

Builds triggered via API, or pushed by users that don't have a CircleCI account are currently broken. We have determined the cause, and are pushing a fix.

Posted Jan 20, 2017 - 00:41 UTC