Connectivity Issues - APAC (R3) region
Incident Report for Dotdigital
Postmortem

Summary of impact:

At approximately 07:10 UTC on Wednesday 25th January 2023, services in our Asia Pacific region (R3) became unreachable due to a Microsoft Azure network outage. Services were fully restored at approximately 09:30 UTC.

Customers using our R3 environment may have experienced some of the following issues:

  • Intermittent access to their Dotdigital account and API
  • Delayed or failed Segment refreshes. This may have had the knock-on impact of preventing some campaign sends.

Customers in our Europe (R1) and North America (R2) regions may have noticed brief periods where accessing Dotdigital was not possible or it was slow to load.

Root Cause:

Dotdigital depends upon the Microsoft Azure cloud which suffered a global network outage impacting all Azure regions. In a preliminary RCA, Microsoft declared a change made to their Wide Area Network (WAN) impacted connectivity between clients on the internet and Azure.

Mitigation:

Microsoft reverted a recent configuration change to their Wide Area Network which allowed us to restore normal service at approximately 09:30 UTC.

Next Steps:

Microsoft are a key technology partner, and we rely upon their cloud infrastructure to host our platform. We make use of many high availability features provided by Azure, but none are able to cope with a global Azure network outage. We’ll work with our partners at Microsoft to better understand the root cause of this outage and the steps being taken to prevent future occurrences.

Posted Jan 25, 2023 - 16:11 GMT

Resolved
Our cloud provider have fully restored service and closed their incident.
Posted Jan 25, 2023 - 11:02 GMT
Monitoring
Dotdigital service is now fully restored and our cloud provider have reversed a recent change which could have triggered this problem. We continue to monitor and will react as events unfold.
Posted Jan 25, 2023 - 09:46 GMT
Update
Due to the nature of the problem with our cloud provider a portion of users may experience problems accessing Dotdigital from their location. In addition we currently have a problem with Segments and CPaaS functionality in our APAC (R3) region.
Posted Jan 25, 2023 - 09:26 GMT
Identified
The connectivity issue with our APAC (R3) region has been identified, and is being caused by an outage with one of our cloud providers who are experiencing issues globally
Posted Jan 25, 2023 - 08:28 GMT
Update
We are continuing to investigate this issue.
Posted Jan 25, 2023 - 07:41 GMT
Investigating
We are currently experiencing connectivity issues that are affecting customers using the APAC (R3) region of Dotdigital
Posted Jan 25, 2023 - 07:40 GMT
This incident affected: Asia Pacific - Dotdigital R3 (Asia Pacific - Web Application, Asia Pacific - API, Asia Pacific - Mail Sending, Asia Pacific - Open and Link Tracking, Asia Pacific - Reporting, Asia Pacific - Surveys and Forms, Asia Pacific - Pages and Forms, Asia Pacific - Transactional Email, Asia Pacific - Contact Imports, Asia Pacific - SMS, Asia Pacific - Email to SMS, Asia Pacific - Integration Hub).