Issue with Campaign Reports
Incident Report for dotdigital
Postmortem

Summary of impact:

On Monday 13th September 2021, we experienced an issue with campaign reports which meant some customers who sent campaigns from the web interface were unable to view their campaign reports. This issue started in our: 

  • North America and Asia Pacific regions at 13:36 UTC.
  • European region at 14:41 UTC. 

We reverted the problem code at 16:07 UTC on 13th September. Campaigns sent through the web interface after this time were not affected.

Campaigns created inside the incident window required further intervention to make the campaign reports visible. We resolved this for the majority of customers at 17:11 UTC on 13th September. However, a small number of customers wouldn’t have been able to access their reports until 17:21 UTC on 14 September (at which point campaign reports had been fully recovered for all affected customer). 

During the affected period, customers clicking on a sent campaign would’ve been redirected to the dashboard, rather than viewing the campaign report. It’s important to note, all emails were still sent and we were able to recover all missing reporting data.

This issue only affected email campaigns (SMS campaigns were unaffected).

Root Cause:

We applied a patch to an internal monitoring tool in order to add additional metadata to an email send that helped us track it’s progress through our sending infrastructure. This clashed with a change deployed to the sending pipeline at the same time and had the unexpected effect of failing to initialize campaign reports.

Mitigation:

Once the issue had been identified, we reverted the change to the internal monitoring at 16:07 UTC on 13th September. We continued to work on mitigating the immediate effect so most customers could access their campaign reports. This was completed at 17:11 UTC on 13th September. 

During the next day (14th September), we continued our work on making reports more accurate. We finished updating affected reports at 14:17 UTC on 14 September.

We identified a small number of affected campaigns the following day that had been scheduled for sending during the incident. We rectified these reports at 17:21 UTC on 14 September.

Next Steps:

We will analyse why our QA process did not detect this fault prior to release and make subsequent improvements to our test cases.

Posted Sep 15, 2021 - 13:48 BST

Resolved
We’re satisfied our initial fix has corrected the problem with Campaign Reports. If you experience any further issues with reporting the please reach out to our Support Team.
We’re going to create a detailed report to explain what happened today. If you look back here in a day or two, you’ll find it attached. Sorry about the interruption this caused today.
Posted Sep 13, 2021 - 17:46 BST
Update
Thanks for your patience. We’re making some real progress on resolving the issue with Campaign Reports. We’ll be back with another update very soon.
Posted Sep 13, 2021 - 17:20 BST
Identified
We're investigating an issue with Campaign Reports. When clicking on campaign reports via the web you may be redirected back to the homepage. Please be assured that the email campaign has been sent but reports will not be accessible.
Sorry if you're affected, but our tech team are working as quickly as possible to resolve the issue and get things back to normal. We'll share another update shortly.
Posted Sep 13, 2021 - 16:53 BST
This incident affected: Europe - Engagement Cloud r1 (Europe - Reporting), North America - Engagement Cloud r2 (North America - Reporting), and Asia Pacific - Engagement Cloud r3 (Asia Pacific - Reporting).