Build System Interruption
Incident Report for CircleCI
Resolved
This incident has been resolved.
Posted Mar 24, 2018 - 02:48 UTC
Monitoring
We have finished re-following all users
Posted Mar 24, 2018 - 02:26 UTC
Update
We are continuing to work on re-following all users to their previously followed projects.
Posted Mar 24, 2018 - 01:38 UTC
Update
We are continuing to work on re-following all users to their previously followed projects.
Posted Mar 24, 2018 - 00:59 UTC
Update
We are continuing to process users and re-follow them to their previously followed projects.
Posted Mar 24, 2018 - 00:14 UTC
Update
We are continuing to work on re-following all users to their previously followed projects.
Posted Mar 23, 2018 - 23:45 UTC
Update
We are continuing to work on re-following all users to their previously followed projects.
Posted Mar 23, 2018 - 23:09 UTC
Update
We have re-followed some users to their projects and are continuing to work on re-following all users to their previously followed projects.
Posted Mar 23, 2018 - 22:40 UTC
Update
We are continuing to re-follow users to projects. If you continue to experience issues with projects not building or being unable to follow a project, please follow the instructions in the previous update to reset your project webhooks.
Posted Mar 23, 2018 - 21:38 UTC
Update
We are currently working on re-following users to projects that they were previously following. If we have not gotten to your project yet you can also repair this issue manually by having your organization's github administrator do the following steps:

1. Remove all CircleCI webhooks and services from your GitHub repo settings.
2. Tell the project to stop building on CircleCI through "Project settings".
3. Add the project back on CircleCI.
Posted Mar 23, 2018 - 20:43 UTC
Update
We are in the process of restoring project access and additionally investigating reports of project configurations not being used on new builds.
Posted Mar 23, 2018 - 20:05 UTC
Identified
Our upstream provider has deployed a fix and we have confirmed it to work. We expect all GitHub users to regain access to their existing projects however we are currently correcting secondary issues that may be occurring due to the initial upstream outage.
Posted Mar 23, 2018 - 19:37 UTC
Monitoring
Our upstream provider has deployed a fix and we are able to read access rights again. We are currently monitoring our build system and are investigating further steps on our end to correct any potential remaining permissions issues.
Posted Mar 23, 2018 - 19:21 UTC
Identified
Our upstream provider has determined the root cause and is currently rolling out a fix for the issue causing GitHub users to have their project access revoked.
Posted Mar 23, 2018 - 19:04 UTC
Update
We are currently working with our upstream providers to determine the issue preventing access to GitHub projects.
Posted Mar 23, 2018 - 18:54 UTC
Update
We have temporarily disabled project access revocation. We continue to investigate the root cause of this problem and will provide more information as it becomes available.
Posted Mar 23, 2018 - 18:36 UTC
Investigating
We are currently investigating an issue causing project access to be revoked, preventing builds from running. We will provide updates when we have further information.
Posted Mar 23, 2018 - 18:13 UTC