At approximately 13:00 BST on Tuesday 25th June 2019 websites for all Engagement Cloud regions became inaccessible from various regions of the Internet.
Our engineers soon isolated the problem to Cloudflare which appeared to be unreachable. Cloudflare are a popular security and CDN product who sit in front of our websites, they also noticed the disruption and made an announcement on their status page.
Cloudflare identified the cause of the outage as a BGP leak which was triggered by a series of mis-configuration events at two US based companies one of which being Verizon. BGP or Border Gateway Protocol is mechanism used by networks to transfer data, it's this exchange which enables individual networks to join to form the Internet. In the event of bad BGP configuration being released it’s possible for traffic intended for one network to be sent to another. In this case data intended for Cloudflare’s network was being routed to a small US company who couldn’t handle the increased load. The same problem affected many other Internet services including Amazon. Cloudflare have posted more information about this incident here.
At 13:40 BST traffic to Cloudflare and our platforms was restored after the network responsible corrected their mistake.
BGP problems are extremely difficult to defend against. It’s an inherent flaw within the systems making the Internet possible which allow a network owner to potentially cause disruption for other networks. We rely upon the larger Internet carriers to monitor for and ignore these mis-configurations and in this case this did not happen.