After a regular application deployment on Wednesday 3rd March 2021, we encountered an issue processing bounces, replies and other inbound emails. There were two specific issues as a result:
There was no impact to outbound messages or transactional emails during this time.
A summarized timeline of events:
Our platform receives several different inbound email types:
They are processed by our inbound mail services and routed to the right parts of Engagement Cloud. We added a new piece of functionality to these inbound mail services in our scheduled application deployment, which had a bug impacting some of those email types. This bug caused some messages not to be saved correctly. This meant we could recover the vast majority of these messages, but a small number couldn’t be recovered successfully (once the bug was fixed).
After we identified the issue, we rolled back to the previous version of the inbound email services to stabilize them. However, because we had accumulated a backlog of unprocessed email during the error period, we made a hotfix to make sure the backlog was processed quickly. Without this hotfix, the time to clear the backlog of unprocessed email would’ve been substantially longer.