We are now out of the double-failure scenario and performance of underlying systems has returned to more normal levels. Systems are still processing backlog from the degraded period and therefore overall user experience is not 100% back to normal yet.
In general, going forward we should see backups execute in a timely manner albeit possibly taking longer than normal due to previous delays. With this additional inflow of data we will expect to see delays in metadata import which will affect the browsing experience of the most recent backups for those affected. We expect to recover from this situation fairly quickly as we process the backlog.
We are monitoring this catch-up phase and will update this incident as we have news to report.
Posted Oct 22, 2020 - 07:54 UTC
Identified
Failures across multiple systems at the same time is causing a performance degradation on backup jobs and related services such as search data imports.
No data is lost and all automatic recovery mechanisms self-activated as intended during the two separate failure events.
Usually such failures go unnoticed by customers as systems dynamically reschedule loads to faster unaffected systems but in this particular case, because of two almost simultaneous failures, all load cannot transfer to systems that are not affected.
We are working on multiple tracks to re-establish usual performance. This issue will be updated until resolved.
Posted Oct 20, 2020 - 08:11 UTC
This incident affected: United States, Washington DC (us-dc) (SaaS Backup).