All backlog has completed processing and performance has stayed excellent throughout the busiest window of the day.
We are closing this incident.
Posted Jan 15, 2021 - 07:52 UTC
Update
The vast majority of backlog has been processed and performance continues to look good even as we are entering a higher load period of the day.
We will continue to monitor the situation but expect the last tough bit of backlog to process to completion over the coming hours.
Posted Jan 14, 2021 - 23:17 UTC
Update
Performance is still looking good as expected and backlogs continue to process quickly.
We continue to follow the situation
Posted Jan 14, 2021 - 16:35 UTC
Monitoring
Two separate actions have been taken:
A software fix that significantly improves our ability to work around performance issues as what we faced today in the US region has been deployed. This fix significantly improved the overall performance of the platform as it allows us to effectively reduce workload sent to temporarily overloaded systems.
Secondly, a hardware device that was causing disruptions to other devices (a so-called "noisy neighbour") was identified and offlined. Corrections such as this are usually fully automatic, but in certain cases automatic systems are unable to identify this situation and manual intervention is required. This correction significantly improved performance on one of the temporarily overloaded systems.
All in all, we see a 10x improvement in processing latency for data operations for the affected users. We are starting to see an improvement on the processing backlog for search data imports. The backlog will take some time to clear though.
It still appears that backup performance was not significantly affected by this.
We will continue to monitor the situation and expect to see the backlog starting to clear over the coming hours.
Posted Jan 14, 2021 - 14:48 UTC
Identified
We are experiencing degraded performance on certain systems in the US region. This is causing a delay in backup data imports into the search engine which may in turn cause a delay between a backup completing and the backup being available for easy search based browsing. This performance degradation will also affect downloads and other data operations.
Backup itself seems largely unaffected by this, but some delays could occur as a result of this degradation.
A fix has been developed to remedy this situation and this fix will be deployed on to the US region in the coming hours.
Posted Jan 14, 2021 - 13:22 UTC
This incident affected: United States, Washington DC (us-dc) (User Interface).