Slow Github Webhooks

Incident Report for CircleCI

Postmortem

At 4:30 PM PST our customers started to notice extreme slowness with all parts of our site and around 4:50 PM PST we declared an incident.

During investigations into the root cause we determined that GitHub was suffering from extreme latency for some of their API's. This latency was having an impact on CircleCI at all levels which translated into site slowness and/or build failures.

GitHub resolved their issue around 5:50 PM PST which then also returned out site back to normal.

Posted Jan 31, 2017 - 19:08 UTC

Resolved

We're no longer experiencing issues with Github's API. Please reach out to support@circleci.com if you have any issues.

Posted Jan 31, 2017 - 01:54 UTC

Update

We're still seeing slow response times from Github's API, however there are signs of improvement. We'll continue to monitor and update again in 20 minutes.

Posted Jan 31, 2017 - 01:42 UTC

Monitoring

We're continuing to monitor slow response times from Github's API, we'll update again in 20 minutes.

Posted Jan 31, 2017 - 01:13 UTC

Investigating

We are noticing slow response times from GitHub webhooks. We are monitoring and will update again in 20 minutes.

Posted Jan 31, 2017 - 00:53 UTC