Frequent backup failures
Incident Report for Keepit
Resolved
Backups have continued to operate as expected since the problem was resolved and by now all connectors should have fully caught up with their backups - any delay experienced at this time will be due to external factors such as SaaS provider throttling or increases in backup set size. We are therefore marking this incident as fully resolved.
Posted Jan 23, 2020 - 12:05 UTC
Update
Backups have been operating as expected since the issue was corrected. We are letting this incident live a bit longer since a small number of customers will still have connectors that are late with backups due to the previous interruption. We expect the last few connectors to have caught up soon.
Posted Jan 22, 2020 - 14:47 UTC
Monitoring
We identified the root cause and corrected it - we are now monitoring the systems to ensure that service is fully restored.

Affected customers will have received notices from the system that backups have failed - but regular backups should now again be scheduled in a timely manner, and all customers should again return to normal operations over the next 24 hours as backups are executed.
Posted Jan 21, 2020 - 11:28 UTC
Update
We have confirmed that the issue is only affecting customers on the EU environment.

Work is progressing on locating the root cause for backups not being executed as they should be.
Posted Jan 21, 2020 - 10:23 UTC
Investigating
We are noticing a much increased occurrence of failed backup jobs and we are investigating the root cause of this.
Posted Jan 21, 2020 - 08:05 UTC
This incident affected: Denmark, Copenhagen (dk-co) (SaaS Backup), United States, Washington DC (us-dc) (SaaS Backup), and Australia, Sydney (au-sy) (SaaS Backup).