Storage performance degradation
Incident Report for Keepit
Resolved
Systems have been operating nominally for several days now and all customers are back to regular backup performance.
Posted Sep 18, 2019 - 07:48 UTC
Update
The environment has been operating at full speed over the weekend and nearly all deferred work has been completed. Customers should generally experience normal performance in all aspects of the platform.

We will be monitoring the situation closely for the rest of today to ensure that we have gotten everyone back to full service.
Posted Sep 16, 2019 - 08:43 UTC
Monitoring
All new capacity expansions have been installed, configured and activated. They are operating as expected.

Processing of delayed backups and deferred work will commence over the weekend. We continue to monitor the situation closely but still expect to return fully to normal operations by Monday.
Posted Sep 13, 2019 - 14:20 UTC
Update
All new systems have been delivered and are being installed in the datacenters. Configuration and enabling of the systems will commence through the day.

We expect to be operating at full capacity again late Friday and for all deferred/late processing to have caught up over the weekend.
Posted Sep 13, 2019 - 09:38 UTC
Update
Several configuration and tuning improvements have been implemented to improve the current levels of performance and we are seeing positive effects from this. The backlog of delayed activity is being processed but we expect some delays on the service for some time still.

We have worked with our suppliers in expediting the planned performance expansion and we expect to finish this already by tomorrow, Friday the 13th.
Posted Sep 12, 2019 - 10:36 UTC
Update
Symptoms affected customers will face during this period will be delayed start of backups, and possibly a change in the connector health status.

All data is still available for download, restore and browsing - we are continuing to work on remediating the situation as quickly as possible. We also work with suppliers to expedite the current capacity expansion.
Posted Sep 11, 2019 - 15:34 UTC
Identified
We are seeing degraded storage performance, especially relating to writing of new data, for approximately one third of our customers on the EU region.

A larger expansion of capacity was planned and is under way with expect completion before the end of next week. We are currently considering options for temporarily working around these issues to increase throughput until the expansion is completed.
Posted Sep 11, 2019 - 13:15 UTC
This incident affected: Denmark, Copenhagen (dk-co) (SaaS Backup).