Insight Data Import Delays - Europe
Incident Report for dotdigital
Postmortem

Summary of impact:

At approximately 06:45 UTC on the 9th December 2019, Insight Data imports started queuing and users experienced delays submitting their data.

We were alerted to this problem at 09:06 UTC on the 10th of December 2019 and began to take corrective measures immediately. We added additional server capacity and by 11:58 UTC the import backlog was cleared and normal import speeds were restored.

Root Cause:

A legacy component dealing with en-queuing insight data imports stopped working. This was due to a locking mechanism that stops this process from running in parallel failed to operate as expected. This led to the platform not processing imports and a backlog building up. We have comprehensive monitoring on our message queues, but as this component used an old proprietary queuing mechanism it was not monitored.

Mitigation:

We restarted the en-queuing process which caused imports to start being processed again. We scaled up the virtual machines responsible for processing imports and increased parallelism of imports to clear the backlog faster.

Next Steps:

We’re adding the missing monitoring to this legacy en-queuing component. We’ll work towards removing the component and replacing it with our standard message queuing solution.

Posted Dec 13, 2019 - 13:43 GMT

Resolved
The import delay with Insight Data in Europe has now been resolved.
We apologise for any inconvenience.
A full RCA will be available in the next few days.
Posted Dec 10, 2019 - 12:08 GMT
Identified
We have increased capacity and queued Insight Data is being processed. Some customers may still experience delays whilst the backlog is cleared.
A further update will be provided in due course.
Posted Dec 10, 2019 - 11:29 GMT
Investigating
We are currently experiencing importing delays with our Insight Data feature. This is currently being investigated and an update will be provided when available.
We apologise for any inconvenience this may cause.
Posted Dec 10, 2019 - 09:58 GMT