Failures of backup jobs
Incident Report for Keepit
Resolved
This issue has been resolved
Posted Apr 12, 2021 - 10:03 UTC
Update
The fix has been deployed across all regions. We will continue to monitor the situation to ensure the fix is effective.
Posted Apr 08, 2021 - 16:37 UTC
Update
Hot fix will be implemented across all regions. Ongoing ZIP downloads and PST file generation may fail. If this influences your data downloads, please start them again.

We will continue to monitor the situation to ensure the fix is effective.
Posted Apr 08, 2021 - 13:55 UTC
Update
After deploying the patch yesterday, that was aimed at fixing the recently discovered issue related to the changes made to Microsoft Graph API, we are confirming that the initial problem has been resolved. We have been closely monitoring the backup progress and can assure you that all backup jobs either completed successfully, or are in progress now, meaning there have been no interruptions so far.
While working on solving this issue, we have determined one more area that changes to Microsoft Graph API have negatively impacted. The error appears on connector configuration step when the users and group lists have to be loaded. Our developers have already made a fix for this case as well, and we are now testing it out. As soon as we have positive test results, we will release it.
We will provide further updates on the progress.
Posted Apr 08, 2021 - 11:51 UTC
Monitoring
The fix has been deployed across all regions. We will continue to monitor the situation to ensure the fix is effective.
Posted Apr 07, 2021 - 17:25 UTC
Identified
We have received multiple complaints from our customers located on US and UK environments about the backups being failed. After checking all our systems we could not locate any problems on our side. That is why on April 6 we had to urgently deploy a hotfix that would allow us to get more detailed logs on specific environments.
After analyzing those logs we were able to detect the root cause of the issue. It appeared that it is related to the latest changes of Microsoft Graph API which Keepit uses for backing up OneDrive, SharePoint, and getting information about groups. So the problem affects rather a small amount of customers on US and UK regions, and it is not present on specific tenants and is not related to the size of the tenant.
Our engineers have now developed a hotfix which should eliminate the problem, and we are getting ready to release it today, on April 7.
We will provide further updates.
Posted Apr 07, 2021 - 10:46 UTC
This incident affected: Denmark, Copenhagen (dk-co) (User Interface, API, SaaS Backup), United States, Washington DC (us-dc) (User Interface, API, SaaS Backup), Australia, Sydney (au-sy) (User Interface, API, SaaS Backup), and United Kingdom, London (uk-ld) (User Interface, API, SaaS Backup).