Global outage accessing Dotdigital
Incident Report for Dotdigital
Postmortem

RCA: Intermittent Access to Engagement Cloud

Summary of impact:

At approximately 06:31 UTC on Tuesday 21st June 2022, our monitoring alerted us to connectivity issues to our platform. During this time, customers experienced issues when trying to gain access to their Dotdigital account or the API. In addition, all inbound activities such as surveys, email and SMS interactions and contact signups would have worked intermittently. 

Root Cause:

We isolated the problem to be with a 3rd party provider, Cloudflare. Cloudflare have since announced the incident was caused by an incorrect network configuration change which disrupted traffic in some of their largest data centers. You can find the full details of the incident on the Cloudflare blog post https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/.

Mitigation:

At 07:29 UTC,  Cloudflare resolved the issues with their network routing. This meant access to our platform and APIs were restored.

Next Steps:

We’ve asked Cloudflare to continue their investigation into this incident and to identify any mitigating steps they can take to prevent future issues.

Posted Jun 22, 2022 - 13:27 BST

Resolved
This incident has been resolved.
Posted Jun 21, 2022 - 09:01 BST
Update
Cloudflare have identified the cause of the problem and applied a fix which has restored service to Dotdigital. All of our systems are now working as normal and we're monitoring the situation.
Posted Jun 21, 2022 - 08:21 BST
Monitoring
Dotdigital is currently experiencing a global outage caused by an incident at Cloudflare. We’re working with Cloudflare to understand the situation and restore services as quickly as possible. During this period it’s likely customers will not be able to access their accounts via our portal or the API, click and tracking data will also work intermittently.
Posted Jun 21, 2022 - 08:20 BST