Paused email sends
Incident Report for dotdigital
Postmortem

Summary of impact:

As part of our fortnightly release, at approximately 08:03 BST on Wednesday 25th September 2019, we made a small update to the way we compile and personalize email campaigns. Shortly after the release customers reported that buttons built by our content editing tool ‘EasyEditor’ were removed from the email content. To prevent the content in emails being delivered incorrectly, we paused email sends across all regions at 08:57 BST.

At 09:19 BST, Transactional Email was restored in all regions.

At 09:51 BST, email sends were temporarily restored but were disabled again at 10:12 after we identified the problem still existed.

At 11:17 BST, the problematic code was rolled back and full service was restored.

Root cause:

At the start of the sending process, a number of changes are made to campaign HTML in order to enable tracking, add personalization, control email rendering in different email clients, etc.

In order to prevent email content being truncated and clipped in some email clients, we made a change to remove unnecessary comments and white space from the HTML source of email campaigns. This code did not correctly exclude ‘buttons’ from this removal process, meaning content containing buttons were unintentionally removed. This wasn’t picked up in our automated or manual testing phases of the release.

Mitigation:

We identified the specific piece of code causing the issue and rolled back the change to a known stable version.

Next steps:

We’re going to:

  • Take further steps to extend our automated and manual tests to pragmatically validate the resulting email source against the original content as supplied by the user.
  • Create a new version of the code and make sure it’s re-tested with a laser-focus on testing on buttons before being redeployed in a subsequent patch or release.
Posted 24 days ago. Sep 25, 2019 - 16:23 BST

Resolved
This incident has now been resolved.
We apologise for the inconvenience this has caused.
A full report will be available in the next few days.
Posted 25 days ago. Sep 25, 2019 - 11:05 BST
Update
We are continuing to work on a fix for this issue.
Posted 25 days ago. Sep 25, 2019 - 10:51 BST
Update
A fix for this issue will be rolled out very soon.
We apologise for the continued disruption.
Posted 25 days ago. Sep 25, 2019 - 10:28 BST
Identified
We have identified the issue are now rolling back to a previous release.
Posted 25 days ago. Sep 25, 2019 - 09:32 BST
Update
We are continuing to investigate this issue.
Posted 25 days ago. Sep 25, 2019 - 09:28 BST
Update
We are continuing to investigate this issue.
Posted 25 days ago. Sep 25, 2019 - 09:17 BST
Investigating
Email sends in all regions have been paused whilst we investigate an issue with buttons not appearing in campaign content.
Posted 25 days ago. Sep 25, 2019 - 09:03 BST
This incident affected: Asia Pacific - Engagement Cloud r3 (Asia Pacific - Mail Sending, Asia Pacific - Transactional Email), North America - Engagement Cloud r2 (North America - Mail Sending, North America - Transactional Email), and Europe - Engagement Cloud r1 (Europe - Mail Sending, Europe - Transactional Email).