Post-Mortem: The Alert That Did Not Fire (At the Correct Severity)

On a recent date, the perimeter experienced an event. Service was, for a period, degraded; instruments held allocations but returned only bad responses, while reporting themselves 100% synced. We wish to be transparent about what happened, to the extent that transparency is available.

Summary

We maintain several automated alerts. In this instance, the relevant alert did not trigger with the correct severity. The event was instead detected by humans, who noticed — as humans will — that nothing worked. We thank them, and have referred their initiative to a Subcommittee.

Timeline

• T+0 — A fix is being deployed now.
• T+1h — The fix ran into some issues with the deploy. We were, candidly, a bit premature with the notification. Apologies.
• T+2h — They are working on the deployment now.
• T+4h — Deployment is going through now; things should be restored shortly. Sorry this took much longer than expected.
• T+∞ — Resolved from our end as well. Thank you.

“Are there not automated alerts when things go wrong?”

Yes. We have several automated alerts set up. Unfortunately, this one did not trigger with the correct severity. We will conduct a post-mortem and change alerts as needed. The post-mortem is, itself, pending, and has been assigned a severity, which did not trigger.

Remediation

Going forward, if you notice an issue with our stack, feel free to reach out directly. We have, in response to this incident, drafted a new alert. It is rigorous, threshold-based, and broadly rectangular. We are confident it will fire at the correct severity, on a future occasion, which we will detect by other means.