Summary of impact:
At approximately 16:52 UTC on Thursday 14th April 2022, we detected issues with self serve Integration Hub integrations in all regions. We could see they were unable to complete their scheduled executions successfully. We resolved the issue with services being restored across all regions at approximately 18:46 UTC on Thursday 14th April 2022.
Customers may have experienced some of the following issues:
Root Cause:
The system hosting the self serve integrations experienced an issue with the data storage systems used to hold the configuration and state. This caused the integrations to be unable to retrieve their configuration reliably and caused them to error during any attempted executions during the outage period.
Mitigation:
The timeline (in UTC) for resolving this issue was:
- 16:52: We detected issues with self serve integrations terminating with errors and started investigations
- 17:12: We established the issues were widespread and across all regions. The storage system used for storing state and configuration was identified as the root cause for the issues
- 17:18: We raised a priority 1 ticket with our provider and continued to work with them to help identify the issue and test resolutions
- 18:46: We were notified by our provider the issue with storage was resolved. We verified this and started checking integrations were functioning correctly
- 20:29: After extensive checks and monitoring the incident, we declared the issue resolved. Services had returned to normal operation at approximately 18:46 UTC
Next Steps:
- Work with our provider to identify the root cause of the storage failure and undertake any appropriate actions to avoid a future incident (In progress)
- Update the heartbeat check to verify the storage is operational for the self serve integrations. We believe this may give us faster detection of a similar issues in the future (Completed 19th April 2022)