Problems with custom from addresses using the .uk TLD

Incident Report for Dotdigital

Postmortem

Summary of impact:

At approximately 16:27 UTC on Wednesday 14th September 2022, a change was made to our DNS glue records. We were alerted to an issue with our custom from addresses using the .uk TLD at 08:39 UTC on Thursday 15th September 2022. We restored services at 16:00 UTC on 15th September 2022.

During this time, customers may have experienced some of the following issues:

Links not opening in email campaigns
Landing Pages on hosted domains not loading correctly
Customized login pages not functioning.

Root Cause:

As part of improvements to our DNS infrastructure, we removed glue records from the dotmailer.co.uk domain as they were no longer required. Our registrar’s system then incorrectly identified our nameservers as missing and updated all domains using nsX.dns.dotmailer.co.uk to dummy records which stopped name resolution working. This update from our registrar also incorrectly updated domains which they didn’t control via Nominet.

Mitigation:

The timeline on Thursday 15th September 2022 (in UTC) for resolving this issue was:

08:50 - We confirmed the issue was related to incorrect NS records and confirmed only .uk domains were affected
09:56 - We started to update all .uk domains we control with the correct NS records
10:10 - Update of NS records completed. This resolved around 70% of the affected domains. However, propagation could have continued for up to 24 hours
10:20 - We discovered domains we don’t control had also been affected
10:23 - We began engaging with our domain registrar
13:41 - Our domain registrar confirmed they had raised the issue with Nominet
15:34 - We took ownership of the dummy nameserver domains
16:00 - We confirmed the .uk domains that were still impacted were now reachable with the dummy nameserver records
16:33 - Our domain registrar started to reverse the addition of the out-of-zone to domains they control
16:40 - We updated the dummy nameserver domains to point to our nameservers. This enabled the faulty sites to function again whilst Nominet looked into the issue

Next Steps:

Our investigations into the issue are still ongoing. We’re working alongside our domain registrar to identify lessons learned and drive process changes to ensure a re-occurrence isn’t repeated.

Posted Sep 16, 2022 - 17:05 BST

Resolved

The previously broken from addresses continue to work and have remained stable since our last change. In rare cases some customers may have incorrect DNS cache which could prevent domains from working properly. This cache is local to the device being used and will clear without requiring further intervention. Root cause investigation continues and we will post further updates here in due course.

Posted Sep 15, 2022 - 18:09 BST

Monitoring

The remaining set of broken from address domains have now been fixed. Work continues with our domain registrar and Nominet to understand the root cause of the problem.

Posted Sep 15, 2022 - 17:06 BST

Update

We are continuing to restore domains that are not held with Dotdigital's domain registrar. Customers specifically affected by this will soon receive a separate communication on how to resolve the problem.

Posted Sep 15, 2022 - 14:51 BST

Update

Custom from addresses which are hosted via Dotdigital's domain registrar have now been corrected and they will begin to operate normally. This set of domains represents approximately 70% of the domains impacted by this problem.

Posted Sep 15, 2022 - 11:20 BST

Identified

The cause of the problem has been identified as an incorrect change to the DNS glue records used by our co.uk name servers. This change has been reverted. However, in response to the glue record change, domains have had their name server setting updated by their registrar to point a non existent hostname. We are now working to restore the correct name server settings in domain registrar.

Posted Sep 15, 2022 - 10:43 BST

Investigating

We are investigating problems with custom from addresses using the co.uk, org.uk and all other .uk domains. This problem will prevent links from working in email campaigns and we advise customers using a .uk domain not to send whilst we work on this problem.

Posted Sep 15, 2022 - 10:20 BST

This incident affected: Asia Pacific - Dotdigital R3 (Asia Pacific - Open and Link Tracking), Europe - Dotdigital R1 (Europe - Open and Link Tracking), and North America - Dotdigital R2 (North America - Open and Link Tracking).