Summary of impact:
Starting at approximately 20:15 UTC on Monday 9th March 2020, we experienced significant delays sending email messages for customers hosted in our European region of Engagement Cloud. The root cause was discovered at 23:20 UTC and the backlog was fully cleared by 02:00 UTC on Tuesday 10th March 2020.
This incident was caused because of a significant attack on signup forms. Our signup forms allow contacts to opt in to email communications. Our newer signup forms come with some spam protection, but some customers using older types of signup forms don’t have this spam protection built-in automatically. In this incident, a spam attack aggressively targeted our signup forms, sending hundreds of thousands of individual opt-in requests within a short time period.
Our signup forms are configured to send an email campaign in response to new signups. Due to the size of the campaign and the volume of requests being received, this overwhelmed the email sending resources configured in our European region.
Our team were alerted to the incident at 22:00 UTC. After an initial assessment, we posted an incident notification to our customers through our status page. We increased resources to our email sending infrastructure which had some positive impact but didn’t have capacity to reduce the backlog.
After analyzing application logs, we spotted the ongoing spambot attack and immediately converted the offending signup forms to a newer version with spam protection enabled. This prevented any additional spam signups and allowed the email backlog to reduce quickly.
We’re really sorry this incident occurred and for any disruption it caused. From here, we’ll: