Alert Deduplication: Stop Alert Fatigue Without Muting Anything

How duplicates become fatigue

Alert fatigue rarely starts with too many problems. It starts with one problem reported too many times. A database connection pool runs dry, every request that fails sends an alert, and within ten minutes the channel holds four hundred copies of the same message. People mute the channel to get work done, and the next, different alert goes unseen.

The usual fixes all lose something. Raising thresholds hides the early signal. Rate-limiting the sender drops data. Muting a channel drops everything. Deduplication keeps every occurrence but changes what you're shown: one incident, with a count, instead of one message per event.

What counts as the same alert

Sigtake builds a fingerprint from three fields of every alert you send:

fingerprint = SHA-256(source | title | severity)

When an alert arrives and there's already an open incident with that fingerprint in the same project, the new one folds into it: the occurrence count goes up, "last seen" moves to now, and the repeat is logged on the incident's timeline. No second incident is created.

A few details decide whether two alerts match:

• The match is exact. DB connection pool exhausted and db connection pool exhausted are different fingerprints, as is a trailing space.
• It's per project. The same alert in staging and production produces two independent incidents, which is usually what you want.
• The team isn't part of it. If two senders with different team_codes send the same source, title and severity, the second folds into the first one's incident and notifies that team.

The ingest response tells you which case happened:

const { meta } = await sigtake.alerts.ingest({
  title: 'DB connection pool exhausted',
  severity: 'critical',
  source: 'orders-api',
  team_code: 'K7Q2XD',
  payload: { pool_size: 20, waiting: 143 },
});

meta.is_duplicate;      // true when it folded into an open incident
meta.occurrence_count;  // how many times this incident has fired

The 2nd, 10th and 100th occurrence

Folding repeats into one incident fixes the list. Notifications need their own rule, or a busy incident would still ping you on every repeat. Sigtake notifies at occurrence 1, 10, 25, 50 and 100, then every 100 after that.

The spacing is deliberate. The first notification tells you something broke. The 10th tells you it wasn't a blip. By the 100th it's still going, and the reminder is what keeps a noisy incident from being forgotten while someone works on it.

Writing alerts that group cleanly

Since grouping comes entirely from source, title and severity, the way you write those three fields is your dedup configuration. The rule is short: the title names the condition, the payload carries the details.

Any ID, number, hostname or timestamp in the title makes every occurrence a new fingerprint, and you're back to one incident per event. Move it to payload, where it doesn't affect grouping.

Use source for the service that's reporting, not the machine. checkout-api groups the same failure across ten replicas into one incident. checkout-api-7f9c gives you ten.

Severity is part of the identity

Because severity is in the fingerprint, the same title sent as high and then as critical opens two incidents. That's useful when the severities describe different conditions: "Disk usage above 80%" at high and "Disk usage above 95%" at critical really are separate problems, with separate urgency.

It works against you when a sender picks severity on the fly for the same condition, for example raising it after a few retries. Pick one severity per condition, and if urgency changes, give the worse state its own title.

When an alert folds, and when it reopens

Folding only happens into an open incident, meaning one that's firing or acknowledged. Once someone resolves it, the chain ends:

• Firing or acknowledged: a repeat folds in and the count goes up.
• Resolved: the next repeat opens a new incident with the count back at 1, and notifies as a first occurrence.

That makes resolving meaningful. If a problem comes back after you've marked it fixed, it deserves a fresh notification rather than a quiet bump on a closed incident. Resolve when the cause is actually fixed, not just when the noise stops.

What you see in Slack, email and the app

Each channel shows a folded incident in the way that suits it:

• Slack: the first occurrence posts one card. Later notifications reply in that card's thread without being broadcast back to the channel, so the channel shows one line per incident. When someone acknowledges or resolves it, the original card updates its color and says who did it.
• Email and Microsoft Teams: one message per notification, so at 1, 10, 25, 50 and 100.
• In the app: each person gets one entry per incident in their notifications. At each threshold that entry is refreshed and marked unread again, rather than stacking identical lines.

The incident itself keeps the full picture: total occurrences, when it was first and last seen, and a timeline entry for every repeat.