Alert Routing by Team: Getting Every Alert to the People Who Own It

Why route by team

The easy setup is one channel that gets every alert. It works until the payments engineer has to scroll past forty infrastructure alerts to find the one about failed charges, and starts ignoring the channel. Routing by team gives each alert an owner: the people who can fix it hear about it, and everyone else doesn't.

In Sigtake every alert belongs to exactly one team. That team decides two things: who is notified (its members) and where (the channels it's connected to).

What a team is

A team has three parts:

• Members: the people who receive its alerts, by email and in the app.
• A team code: six characters, such as K7Q2XD, generated when the team is created and shown on its page. It's how an alert names its team, and it doesn't change.
• Projects: the environments it covers, such as production and staging. A team can cover several.

Teams live at the account level, not inside a project. The same "Payments" team can own alerts in production and staging, and each project gives it different channels. That split is what lets staging alerts go to a quiet channel while production alerts go to the loud one, without maintaining two teams.

How an alert finds its team

Where the team comes from depends on what created the alert:

• Alerts you send through the API or the SDKs carry a team_code. It's required, so there's no such thing as an unowned alert from your code.
• Uptime checks and signal monitors are each assigned to a team when you create them, and their alerts belong to that team.

await sigtake.alerts.ingest({
  title: 'Payment provider returning errors',
  severity: 'critical',
  source: 'payments-worker',
  team_code: 'K7Q2XD',   // the Payments team
});

Because the team is part of the alert, routing is decided by whoever sends it. Different parts of one service can report to different teams: a failed charge to Payments, a full disk on the same host to Infrastructure.

Teams and projects

A team only works in the projects it has been added to. Each API key belongs to one project, so when an alert arrives Sigtake checks that its team is assigned to that key's project. If it isn't, the alert is rejected with TEAM_NOT_IN_PROJECT rather than accepted and routed nowhere.

The same rule applies when creating monitors: you can only pick teams that are part of the monitor's project. A team that isn't in any project yet shows a warning on its page, because nothing can route to it until an admin adds it to one.

Who receives what

When an alert notifies, its team's members are reached in up to three ways:

• In the app, always. Every member gets the alert under the bell icon.
• By email, if the team has an email channel in that project.
• In Slack or Microsoft Teams, if the team has one of those channels in that project. These post to the shared channel rather than to individuals.

Channels are covered in detail in the channels guide. The setup itself, step by step, is in how to set up alert notifications.

Who can change what

Team setup is split between roles so day-to-day changes don't need an admin, while anything that changes where alerts go does:

Structuring teams on a small company

Model teams on who fixes things, not on the org chart. A few patterns that hold up:

• Start with one team per area of ownership, such as Platform, Payments and App. If two teams would always contain the same people, make them one.
• Put a person in every team whose alerts they should hear about. Membership is the only way to receive a team's email and in-app alerts, so the person who handles both payments and infrastructure belongs to both.
• Use projects for environments, not teams. "Payments" covering production and staging, with quieter channels in staging, beats a separate "Payments Staging" team.
• Keep a catch-all team only as a stopgap. It's fine while you're getting started. Once alerts pile up in it, split it by the people who actually respond.