At approximately 14:50 UTC on Monday 27th July 2020, 1 of our primary database servers was experiencing severe blocking on one of our client databases. This caused application timeouts and errors as a result. We restored services at 16:00 UTC on Monday 27th July 2020. Some customers may have experienced:
A maintenance process ran on one of our largest client databases. One of the tables within this database was not configured correctly. When the maintenance process ran it was attempting to sort/order half a billion data rows. To do this our back-end database system had to block access to the table while it carried out its maintenance. This had a knock on effect of blocking some of our most important tables.
We located the process that was running on the server causing the blockage. We cancelled the maintenance process at 15:10 UTC. We had to wait for our back-end database system to rollback the changes to ensure the table was in a consistent state, this took approximately 50 minutes