Summary of impact:
At approximately 16:27 UTC on Wednesday 14th September 2022, a change was made to our DNS glue records. We were alerted to an issue with our custom from addresses using the .uk TLD at 08:39 UTC on Thursday 15th September 2022. We restored services at 16:00 UTC on 15th September 2022.
During this time, customers may have experienced some of the following issues:
Root Cause:
As part of improvements to our DNS infrastructure, we removed glue records from the dotmailer.co.uk domain as they were no longer required. Our registrar’s system then incorrectly identified our nameservers as missing and updated all domains using nsX.dns.dotmailer.co.uk to dummy records which stopped name resolution working. This update from our registrar also incorrectly updated domains which they didn’t control via Nominet.
Mitigation:
The timeline on Thursday 15th September 2022 (in UTC) for resolving this issue was:
Next Steps:
Our investigations into the issue are still ongoing. We’re working alongside our domain registrar to identify lessons learned and drive process changes to ensure a re-occurrence isn’t repeated.