This incident has been resolved. Thank you for your patience in this matter.
Posted Sep 30, 2022 - 00:32 UTC
Monitoring
The test results processing pipelines is starting to recover. We are caught up to builds that ran within the last 15 hours and expect to be fully caught up in 2-4 hours.
Posted Sep 30, 2022 - 00:00 UTC
Update
We have deployed both changes to speed up test result processing. We are now deploying changes to remove unprocessable events from the queues.
Posted Sep 29, 2022 - 22:10 UTC
Update
The issue with the Insights web UI has been resolved. Engineers are also monitoring metrics for the fix to test insights processing, once things have stabilized we will complete the rollout of the second update, at which point we believe that processing will return to normal, and we will begin to work through the backlog.
Posted Sep 29, 2022 - 18:58 UTC
Update
Due to an unrelated issue with a back-end database, the Insights UI is currently unavailable. Engineers are currently working on remediation.
Posted Sep 29, 2022 - 18:35 UTC
Update
We are working on the second of two updates at which time we expect to see accelerated processing of the backlog
Posted Sep 29, 2022 - 17:55 UTC
Update
Insights data for tests (flaky tests, slowest tests, most failed, etc) is currently delayed by approximately 2 days. Historical data remains visible and test data has not been lost. We are in the process of deploying multiple changes to accelerate processing and will update as we start to see the delay time reduce.
Posted Sep 29, 2022 - 17:10 UTC
Update
We are continuing to work on a fix for this issue.
Posted Sep 29, 2022 - 16:46 UTC
Identified
We have identified an issue with Insights (affected may notice that the data shown for flaky tests, slowest tests, most failed tests, etc. are delayed). A remediation is currently in progress
Posted Sep 29, 2022 - 15:46 UTC
This incident affected: CircleCI UI and CircleCI Insights.