Regular application update (12th February 2020)
Scheduled Maintenance Report for dotdigital
Postmortem

Summary of impact:

At approximately 07:30 UTC on Wednesday 12th February 2020, we deployed an update which contained a change to a frequently used regular expression to validate email addresses. On specific data sets, this new regular expression caused performance degradation in many Engagement Cloud systems.

Most significantly, we saw service disruption and delays in contact imports, SMS sends and reporting. The impact and resolution times varied slightly depending on region:

European region:
Services returned to normal at 13:20 UTC apart from reporting data which was delayed for a subset of customers until approximately 22:00 UTC.

North American region:
Email sending was delayed between 23:00 UTC and 01:00 UTC on Thursday 13th February.

Root Cause:

The original regular expression to validate email addresses was found to let a small number of invalid, non RFC compliant addresses enter the Engagement Cloud contact database. This bug was addressed and a new regular expression was developed to prevent more malformed addresses. However, the new regular expression executed slower than the previous when certain inputs were used.

Mitigation:

At approximately 11:00 UTC, we discovered the poorly performing regular expression and work began to restore it to its previous version.

We updated the most affected services first, which reduced CPU load across our systems and restored normal operation.

Next Steps:

The regular expression was logically correct and we used defensive techniques such as unit tests to manage the change. But, the change only suffered performance problems on certain inputs. In future, we’ll run through a more rigorous stress test when amending critical regular expressions and use a far larger set of test data.

Posted Feb 13, 2020 - 17:11 GMT

Completed
The SMS feature and contact import issues have been resolved. We apologise for any inconvenience. A full RCA will be available in the next few days.
Posted Feb 12, 2020 - 14:25 GMT
Verifying
Contact imports now appear to running normally. We will continue to monitor and will update again if necessary.
Posted Feb 12, 2020 - 11:27 GMT
Update
We have made a change and are seeing improved performance on our SMS feature. We are still monitoring contact imports and will update you in 1 hour.
Posted Feb 12, 2020 - 10:58 GMT
Update
Our regular application deployment is complete, but we're seeing degraded performance on contact imports as well as the SMS feature.

We are investigating this and will update in due time.
Posted Feb 12, 2020 - 08:44 GMT
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Feb 12, 2020 - 07:30 GMT
Scheduled
Our next regular application update is scheduled for 12th February 2020 at 07:30 GMT.

Regular updates typically bring new features, bug fixes and performance gains.

During the update, sends and related services may run slower than usual. Note that our SMTP endpoints may also go offline during this time.

After the release has been completed you can find the latest set of release notes at https://support.dotmailer.com/hc/en-gb/sections/202602098-Feature-releases

Local regional times for this application update are as follows:
02:30 - 03:30 EST (e.g. New York)
23:30 - 00:30 PST (e.g. San Francisco)
18:30 - 19:30 AEDT (e.g. Sydney)
Posted Feb 05, 2020 - 10:04 GMT
This scheduled maintenance affected: North America - Engagement Cloud r2 (North America - Web Application, North America - API, North America - Mail Sending, North America - Open and Link Tracking, North America - Reporting, North America - Surveys and Forms, North America - Landing Pages, North America - Transactional Email), Asia Pacific - Engagement Cloud r3 (Asia Pacific - Web Application, Asia Pacific - API, Asia Pacific - Mail Sending, Asia Pacific - Open and Link Tracking, Asia Pacific - Reporting, Asia Pacific - Surveys and Forms, Asia Pacific - Landing Pages, Asia Pacific - Transactional Email), Europe - Engagement Cloud r1 (Europe - Web Application, Europe - API, Europe - Mail Sending, Europe - Open and Link Tracking, Europe - Reporting, Europe - Surveys and Forms, Europe - Landing Pages, Europe - Transactional Email), and Global - Website, Global - Login Page, Global - Image Hosting.