Summary of impact:
At approximately 08:43 UTC on 10th August 2022, we were alerted of an issue with images not loading within Easy Editor in our EU Region. We restored services at 10:39 UTC on 10th August 2022.
Root Cause:
Images within Easy Editor are served via a dedicated website. This website acts like a proxy and fetches images to be displayed from our image store. It also provides image resizing functionality. This website recently went through a .NET version upgrade and had been running successfully in production for 6 days prior to this incident. However, on 10th August the website periodically became unable to fetch images from our image store and errors occurred. After analysis we could see application threads were being blocked by a third party component used to record application errors. This starved the website of resources and prevented it from relaying images. Investigation continues but it seems likely that this situation was created as result of the .NET upgrade.
Mitigation:
The timeline for resolving this issue (all times in UTC):
Next Steps:
We are currently investigating thread exhaustion issues which have occurred since the .NET upgrade. These could be related to an incompatibility with our error logging component or possibly a combination of this and a change in behaviour of HTTP connection pooling.