Apple hard bounce suppression issue
Incident Report for dotdigital
Postmortem

Summary of impact:

On Monday 26th April 2021 at approx. 11:00 UTC, Apple experienced a partial outage with their mailing system. This outage had a knock on effect meaning some emails sent to icloud.com, me.com and mac.com email addresses may have incorrectly hard bounced, been subsequently suppressed in customer accounts and added to the Global Suppression List (GSL). Apple resolved the issue and normal services resumed at approx. 16:30 UTC.

Root Cause:

During the stated time period, Apple experienced a problem which led to around 17% of emails to valid email addresses hard bouncing. The bounce error received was: 550 5.1.1 <*>: user lookup success but no user record found. As this is a hard bounce error, it caused email addresses to be suppressed. In addition, these email addresses were inadvertently added to the Global Suppression List (GSL). This meant if customers ran the GSL and applied hard bounce suppressions, these ‘valid’ email addresses would’ve been unsubscribed.

Mitigation:

Once we were notified about the problem, the timeline for resolving the issue was:

  • 28th April: We began a full investigation into the impact and began resolving the issue
  • 29th April: We continued working on a solution
  • 30th April: We unsuppressed all affected hard bounced icloud.com, me.com and mac.com email addresses in customer accounts and removed them from the GSL.

Next Steps:

We’re going to conduct some work to understand the measures we could introduce to minimize the impact we experience when mailbox providers have issues like this.

Posted Apr 30, 2021 - 16:38 BST

Resolved
On Monday 26th April 2021 from approx. 11:00 until 16:30 UTC, Apple experienced an outage with their mailing system. This outage meant emails sent to icloud.com, me.com and mac.com email addresses may have incorrectly hard bounced and been subsequently suppressed in customer accounts. We’re currently working to mitigate the issue and unsuppress the affected icloud.com and me.com contacts in customer accounts. We’ll provide another update on our progress on Thursday 29th April.

-- Update: 29th April 2021 16:15 UTC --
We're continuing to work on unsuppressing the affected icloud.com, me.com and mac.com contacts in customer accounts. We're making good progress and will post another status update on Friday 30th April.

-- Update: 30th April 2021 16:00 UTC --
We've successfully unsuppressed all affected icloud.com, me.com and mac.com email addresses in customer accounts and removed them from the GSL. Sorry this issue occurred and for any disruption it caused. We're currently writing a report to explain the issue in more detail, check back soon to see the report.
Posted Apr 26, 2021 - 11:00 BST