The alert we decided not to send

Sixty-seven emails. Sixty-six days. One duplicate <loc> line in a sitemap, and not one of those sixty-seven messages said anything the first one hadn't already said.

That was our own inbox, from our own product, about our own site. The check was working perfectly. It found a real problem, correctly, every single time it ran. That was the bug.

Two kinds of bad news

A monitoring system deals with two fundamentally different things, and most of them treat both the same way.

Availability. The site is unreachable. The certificate is dead. The heartbeat stopped. Something is broken right now and someone needs to get out of bed. In our code these are the checks that answer true to isAvailability() — uptime, SSL, DNS, cron heartbeats, agent heartbeats, application health, TCP, ping. They are the only ones allowed to open an incident.

Advisory. A sitemap has a duplicate entry. A certificate chain is missing an intermediate. A security header is absent. Three pages have broken links. All real, all worth knowing, none of them worth a phone call at 3am. These cap out at warning and never open an incident.

We had that split from the start. It wasn't enough, because it answers how loud but not how often.

Why a warning repeats forever

An advisory condition doesn't flap and it doesn't fix itself. It just sits there being true.

Your check runs on a schedule. Every run rediscovers the same finding, faithfully. If the notification is wired to the finding, you get one message per run for as long as the condition holds. A duplicate line in a sitemap nobody had time to fix produced a message a day for two months.

Worse than the volume is what the volume does to you. By email forty you have a filter. By email sixty the filter has a folder. And the day something genuinely new arrives on that channel, it lands in the folder.

The fixes that don't work

Rate limiting. "Maximum one of these per hour." It delays the first alert, which is the only one that mattered, and it still repeats forever, just more politely.

Deduplicating on content. Hash the finding, suppress the duplicates. This is closer, but it fails in the direction that hurts: a certificate at fourteen days and the same certificate at one day produce different content, so they both send — and two findings that hash identically get suppressed forever, including after you fixed and re-broke the thing.

Digest everything. Roll advisories into a weekly summary. Now the certificate expiring on Tuesday shows up in Friday's email.

Each of these treats the notification as a property of the event. The fix was to make it a property of the episode.

Episodes, not events

A condition that stays true is one episode, however many times you observe it. So the gate remembers, per check and per event type, the severity level it last told you about:

• No memory for this event? This is the start of an episode. Send it.
• Memory at the same level or higher? Same news, told twice. Stay quiet.
• Memory at a lower level? It got worse. Send it, and remember the new level.
• The condition reports level zero, meaning resolved? Forget the episode entirely, send nothing.

That last rule is what keeps the gate from becoming its own bug. A clean result clears every episode on that check, so the next occurrence alerts fresh instead of being swallowed by a months-old memory of a problem you fixed in June.

The escalation rule is what keeps it useful. A certificate warning at fourteen days, seven days, three days and one day is four steps up a ladder, not the same news four times, so all four break through. The sitemap's duplicate <loc> has exactly one level, so it gets exactly one email — and then silence until you fix it or it comes back.

The alert that lied about why it was sent

While we were in there, we found a worse one.

An SSL check reports warning for several different reasons: the certificate is expiring soon, the chain is incomplete, an unnecessary root is being served, the signature is SHA-1, the thing is self-signed. Five conditions, one status.

The "certificate expiring soon" notification was firing for all five. So you could get an expiry warning about a certificate with eight months left on it, because its chain was missing an intermediate. And because the expiry threshold you configured only governs the expiry condition, that notification also ignored your threshold completely.

Wrong reason and wrong timing, in a message that looked exactly like a real expiry warning. Now the expiry notification fires only when the expiry check has actually crossed the days you set. The chain problem still shows up as a warning in the interface. It just doesn't pretend to be something else.

Flapping is the same problem wearing different clothes

A service that goes down, up, down, up doesn't need an alert per transition. It needs one alert that says: this thing is oscillating.

So flapping gets the same treatment. One notification per flapping episode, and the normal incident and recovery messages are suppressed while it lasts. When it settles, you get told it settled. A service that bounced eleven times in an hour is one piece of news, not twenty-two.

What the gate deliberately doesn't do

It doesn't merge alerts across tenants. The same URL monitored by two different customers produces two incidents, minutes apart, and both are correct. Different people are on call. Different escalation policies apply. One customer acknowledging an incident must never silence another customer's page. The staggering looks like a bug in a support ticket and is the only defensible behaviour.

It doesn't make silence mean "fixed". This is the real cost and it's worth stating plainly. Once an episode is open you are told once. Delete that message and nothing will remind you until the condition escalates a step or clears. For an advisory finding we think quiet is the right default. For an outage it would be indefensible, which is exactly why availability checks don't go through this path at all.

The one we didn't send

The gate shipped, and the sitemap finding went from a message a day to a message. Then nothing, until it was fixed.

That's the whole feature. Sixty-six emails not sent, so that the sixty-seventh piece of genuinely new information has somewhere to land.

Most alerting tools compete on how much they can tell you. The harder engineering problem, and the one that decides whether anybody still reads your notifications in month three, is deciding what not to say.