Resolved -
This incident has been resolved. Thank you for your patience in this matter.
Sep 30, 00:32 UTC
Monitoring -
The test results processing pipelines is starting to recover. We are caught up to builds that ran within the last 15 hours and expect to be fully caught up in 2-4 hours.
Sep 30, 00:00 UTC
Update -
We have deployed both changes to speed up test result processing.
We are now deploying changes to remove unprocessable events from the queues.
Sep 29, 22:10 UTC
Update -
The issue with the Insights web UI has been resolved.
Engineers are also monitoring metrics for the fix to test insights processing, once things have stabilized we will complete the rollout of the second update, at which point we believe that processing will return to normal, and we will begin to work through the backlog.
Sep 29, 18:58 UTC
Update -
Due to an unrelated issue with a back-end database, the Insights UI is currently unavailable. Engineers are currently working on remediation.
Sep 29, 18:35 UTC
Update -
We are working on the second of two updates at which time we expect to see accelerated processing of the backlog
Sep 29, 17:55 UTC
Update -
Insights data for tests (flaky tests, slowest tests, most failed, etc) is currently delayed by approximately 2 days.
Historical data remains visible and test data has not been lost.
We are in the process of deploying multiple changes to accelerate processing and will update as we start to see the delay time reduce.
Sep 29, 17:10 UTC
Update -
We are continuing to work on a fix for this issue.
Sep 29, 16:46 UTC
Identified -
We have identified an issue with Insights (affected may notice that the data shown for flaky tests, slowest tests, most failed tests, etc. are delayed). A remediation is currently in progress
Sep 29, 15:46 UTC