EWS backup update
Incident Report for Keepit
Update
Backups have been executing timely for multiple days; we are monitoring the situation across all regions.
Posted Apr 25, 2024 - 11:59 UTC
Update
Backups execute timely and we are monitoring the situation across all regions.
Posted Apr 23, 2024 - 11:08 UTC
Update
Backups execute timely and we are monitoring the situation across all regions.
Posted Apr 21, 2024 - 07:22 UTC
Update
EWS backups have improved and we are monitoring the situation across all regions.
Posted Apr 19, 2024 - 17:45 UTC
Monitoring
Archive mailboxes are now being backed up again and data transport rates have normalized.
Posted Apr 18, 2024 - 18:40 UTC
Update
We are continuing to work on a fix for this issue.
Posted Apr 17, 2024 - 11:34 UTC
Update
The situation is now improving across many regions. Archive mailboxes are still backed up at a lower priority.
Posted Apr 17, 2024 - 05:46 UTC
Update
We are seeing an even increased inflow of EWS data and are continuing to work with Microsoft to understand and solve the issue
Posted Apr 16, 2024 - 14:22 UTC
Identified
Keepit uses the Microsoft EWS protocol to read some data types. Microsoft has recently made a change in their back-end systems that has increased the volume of data EWS is returning for certain mailbox queries by a factor of 10-100x. We are working closely with Microsoft to help them understand the impact of this change, and hopefully revert it. Other data protection vendors are also affected. Temporarily, we have lowered the priority assigned to backup jobs of archive mailboxes to ensure that primary backups continue to run in a timely way. We will provide ongoing updates here as our investigation and remediation continue.
Posted Apr 12, 2024 - 05:48 UTC
This incident affects: Denmark, Copenhagen (dk-co) (SaaS Backup), United States, Washington DC (us-dc) (SaaS Backup), United Kingdom, London (uk-ld) (SaaS Backup), Germany, Frankfurt (de-fr) (SaaS Backup), Canada, Toronto (ca-tr) (SaaS Backup), and Switzerland, Zurich (ch-zh) (SaaS Backup).