At 1429 UTC our OS X data center vendor experienced a network outage which caused our OS X fleet to stop taking jobs. This reduced our OS X fleet capacity tremendously and we immediately started to get a backlog of jobs.
At 1448 UTC the network was restored and we immediately started to drain the job backlog and we cleared the backlog at 1530 UTC.
We considered the incident as resolved at 1649 UTC.