Summary of impact:

At approximately 15:45 UTC on 14 June 2021, we saw intermittent errors when applications attempted to resolve hostnames via DNS. DNS lookups became reliable again at 23:39 UTC.

During this period, services remained online and the majority of workloads completed successfully. However, some customers may have noticed intermittent errors across all aspects of their account including the API, sends, imports and Programs.

Root Cause:

After investigation, we found DNS errors were occurring in applications hosted on a particular server. We’re continuing our work to understand why this server developed a fault. As all our applications are hosted on multiple servers, all other instances continued to work as normal.

Mitigation:

Our timeline for rectifying this issue was (times stated in UTC):

• 21:23: We discovered a series of campaign sends failing to complete successfully
• 21:29: We started our investigations and narrowed the problem to DNS resolution
• 22:09 We increased capacity to our DNS servers as metrics showed some unusual workload patterns. Afterwards, we performed rolling restarts of all applications to clear the DNS cache. However, the errors continued
• At approximately 23:30, we correlated the applications experiencing DNS errors to a single server and by 23:39 all applications running on the faulty server were migrated. At this point, customers would’ve stopped seeing errors and normal service resumed.

Next Steps:

Our applications are hosted on a cloud native service which aims to provide a reliable hosting environment. However, during this incident a component in this environment failed to work as expected. In conjunction with our cloud supplier, we’ll continue an investigation to identify the events leading to the server failure. Our current action items are:

• Look to include the host metadata in our application logs. This would allows us to monitor the error rates per node and better understand the health of the node.
• Although not directly impacting, all services that need to resolve external hostnames should use our approved list of external DNS resolvers. This helps to mitigate DNS pressure on our core Active Directory DNS service