At approximately 14:34 UTC on Friday 9th July 2021, customers in our APAC region may have experienced delays in sending emails and SMS. We restored services to normal at 16:26 UTC.
This was caused by a 3rd party network related issue at our colocation data center in Sydney. We were forced to pause email and SMS sends in our APAC region in order to prevent any data loss.
*Update Tuesday 13th July 2021: The RCA from our third party supplier is as follows:
*At 14:35 UTC on Friday the 9th of July, the data center in which your solution is hosted from suffered an upstream loss of network connectivity.
The cause was a fault in the redundant switching stack of our upstream network vendor, when a Layer 2 software process caused all network paths to stop forwarding network traffic.
Upstream network engineers arrived on site and reset the devices to rectify the issue at 15:49 UTC and services came back online around 14:08 UTC
The root cause of the issue is still under in depth investigation by our upstream vendor and their equipment supplier, and a likely outcome of this will be a scheduled configuration change to mitigate the issue or upgrade the underlying device firmware to affect the same outcome.
We deeply apologise for any loss that occurred as a result of this outage. We will be discussing the outcome of the above investigation ongoing until we are mutually comfortable the underlying issue has been properly mitigated from happening again.
A summarized timeline of events (all times in UTC):