Summary of impact: At approximately 01:05 UTC on 16th November 2020, customers’ experienced issues sending transactional emails using TLS, this was due to an expired SSL/TLS certificate. We restored services at 03:01 UTC on 16th November 2020. During this time, a subset of our customer's transactional emails were not being sent.
Root Cause: This issue was due to an expired SSL/TLS certificate used by our SMTP service. The expiring certificate was replaced across our infrastructure, in advanced of the expiry date, with the exception of our SMTP service, which was configured via an alternate method. This deviation required a separate deploy of the SMTP services and was missing from our standard operating procedures relating to certificate renewal and therefore was subsequently missed.
We apologise in for any inconvenience this may have caused.
Mitigation: The timeline for resolving this issue was (timestamps in UTC):
- Incident was reported at 01:15 - Issue was identified at 01:57 - A patch for the issue was built and released at 02:42 - After testing, incident was fully resolved at 03:01
Next Steps: In order to prevent this issue from re-occurring, we’re going to update our certificate management standard operating procedure to incorporate our SMTP service.