Summary of impact:

From approximately 11:00 UTC on 29 July 2021, we had reports of intermittent user login failures in our North America (R2) and Asia Pacific (R3) regions. We were able to replicate this and could confirm our Europe (R1) region was unaffected.

As a precautionary measure, we paused message sending and services in all regions until we were able to confirm no data was being lost and could pinpoint the exact issue.

Root Cause:

After investigation, we found our partner Cloudflare (CF) had implemented a rollout of Edge Side Code (at approx. 08:00 UTC) across their platform. As it propagated globally, it affected the routing of our platform and white-label domain addresses throughout their network. Cloudflare later found many of their global Edge nodes were no longer able to run the Edge Side Code required.  Cloudflare posted information on their status page.

Mitigation:

Our timeline for rectifying this issue was (times stated in UTC):

• 11:00: We received reports from a small group of customers who were experiencing login errors in R2/R3 regions.
• 12:00: We reviewed all platform audit logs, SQL and Service Bus configurations for any issues or errors
• 13:00: We confirmed our staging environments are experiencing the same intermittent login redirection to production
• 13:50: We paused sending services in R3 to avoid any risk of data loss
• 14:00: We rolled back of our webapps in R2/R3 was completed to rule out code related issues
• 14:30: We purged the Cloudflare webapp domain cache to ensure this was not a caching issue
• 15:15: We paused all services in all regions except for web services
• 15:30: We conducted tests locally and on our staging environments which circumvented our Cloudflare services to confirm the third party partner issue
• 16:00: We contacted Cloudflare support to help identify a custom hostname issue with one of our domains
• 16:30: We brought R1 services back online once it is confirmed this region is unaffected
• 17:00: Cloudflare support engage with us in tracing the root cause

• 17:30: 

  • We made DNS changes to R2 and R3 to circumvent Cloudflare for some of our public endpoints. At this point, we saw a significant drop in error rates. 
  • We brought services back online in R2 and R3
  • We continued to monitor dropping error rates in all regions

• 18:30: 

  • We replayed the same DNS changes to R1 endpoints
  • We continued to see error rates continue to drop in all regions as we waited for propagation

• 21:00: 

  • Cloudflare confirmed a rollout of updated Edge Side Code had not been fully completed across their network
  • They update their status page
  • We could no longer see the error rates we had seen from earlier in the day

• 00:00: Cloudflare confirms the issue has been resolved at their end.

Next Steps:

Cloudflare allows for our Branded and Custom From Addresses (CFA) to each have different origin servers based on the region they are created in. This information is stored as part of the metatdata of the CFA record in Cloudflare. 

Cloudflare use Edge Side Code (ESC) on their global network to intercept traffic from these CFA domains and make the necessary adjustments to the origin server routing decision. In the event that a CFA metadata does not contain an origin server or this ESC code does not execute, a fallback option is used, which in our case directs to R1 origin servers.

We understand from Cloudflare that as part of upgrades to their services, new metadata fields are now available that are now better served by their ESC and would remove the dependency we have for them to maintain our existing configuration. We’ve already engaged with their solution team on how to best migrate our existing CFA origin servers across to this in a way that’s transparent to our customers and will continue to deploy those changes.  

We’re sorry this incident occurred and we’re grateful for your patience while we worked on resolving it.