Cron Job Monitoring: Catch the Jobs That Fail Silently

Cron has no idea what's supposed to happen. It starts a command at the time you gave it and forgets about it. If the command never starts, there's no exception to catch, no error line in a log and no failed request for an uptime check to see. Error tracking, log alerts and website monitors all watch for something going wrong. A job that doesn't run is nothing happening, and nothing happening looks exactly like everything being fine.

The defense is to stop waiting for an error and start expecting a signal. That's what heartbeat monitoring does, and it's also called a dead man's switch: the job checks in when it runs, and the missing check-in is the alert.

The failure that leaves no trace

Scheduled jobs disappear for boring reasons, and almost none of them produce an error anyone will see:

• The crontab didn't survive a change. A server is rebuilt from an image, a container replaces a VM, a config management run overwrites the file. The job is simply no longer scheduled.
• Cron itself isn't running, or the machine was off, rebooting or scaled down at the minute the job was due.
• The environment changed under it. Cron runs with a minimal PATH and no login shell; an upgrade moves a binary, a credential expires, a mounted volume is missing. The script exits early.
• It exits 0 without doing the work, because an empty input or a skipped branch looks like success.

Then there's the second kind: the job runs, but hangs forever on a lock or a network call, or fails with a non-zero exit code. Cron's only built-in way to tell you is to mail the output to the local user through MAILTO, an inbox that on most servers nobody reads, if mail is even configured.

A real example: backups nobody knew were failing

In January 2017 an engineer at GitLab accidentally deleted data from the primary production database. The team turned to the backups. In GitLab's own postmortem: "The S3 bucket was empty, and there was no recent backup to be found anywhere."

The backup job did run. But "a difference in major versions results in pg_dump producing an error, terminating the backup procedure." The job reported the error the way cron jobs do, by email, and "DMARC was not enabled for the cronjob emails, resulting in them being rejected by the receiver." So, the postmortem says, the team was never aware of the backups failing, "until it was too late." GitLab.com was down for about 18 hours, and the incident affected roughly 5,000 projects, 5,000 comments and 700 new user accounts.

The failure was visible, once, inside the job. The warning travelled through a channel that also failed, and nothing expected a success. A check that waited for "backup finished" would have noticed the first night it didn't arrive.

Quiet failures are expensive when they finally surface. More than half of the operators in Uptime Institute's annual outage survey put their most recent significant outage above $100,000. That figure covers outages of every kind, not just cron, but a missing backup is the kind of problem that turns a bad day into one of those.

How a heartbeat works: invert the question

Instead of asking "did anything fail?", a heartbeat asks "did the thing I expected arrive?" Each job gets its own private URL. The job requests it when it runs, and Sigtake knows when the next request is due. If it doesn't arrive in time, that's an incident.

Two settings define "in time". The schedule is when the job runs: an interval ("Runs every 1 day") or a cron expression with its timezone (0 3 * * * in Europe/Madrid). The grace is how late it may be before anyone is told. When the schedule plus the grace passes without a ping, the heartbeat goes Down and an alert opens: Heartbeat down: nightly-backup.

The alert is an ordinary Sigtake alert, high severity. It goes to the heartbeat's team, through that team's channels in the project (Slack, Microsoft Teams, email) and to every member's in-app bell, the same routing by team as any other alert. Its identity is the heartbeat itself, not its title, so renaming the heartbeat mid-incident doesn't split it into two, and repeated misses stay one incident instead of stacking up.

A new heartbeat starts as "Waiting for the first ping…". If that first ping never arrives within one schedule plus the grace, it alerts too: a heartbeat that was created and never wired up is the same silent failure, earlier.

What a heartbeat catches (and what it can't)

That last row is why the ping belongs at the very end, after whatever check proves the work happened: the backup file exists and isn't empty, the import counted its rows. A heartbeat is only as honest as the moment you send it.

Set one up

1. Open Heartbeats in the app and use New heartbeat. Give it the name you want to read in an alert, choose Interval or Cron expression (with its timezone), set the Grace, and pick the team. Operators and admins can create heartbeats.
2. Copy the line the panel shows. It's built from your heartbeat's real URL and schedule.
3. Run it once. The panel switches from "Waiting for the first ping…" to Up as soon as the ping lands, so you know the line works before you rely on it.
4. Prove the alert reaches someone: pause the job, or point a throwaway heartbeat at nothing, and let the deadline pass.

The team needs a channel in the project for the alert to leave the app. If that isn't set up yet, start with setting up alert notifications for a team; the full options are in the docs.

The crontab line, and why it's written that way

This is the line the panel recommends. It pings /start, runs the job, then reports the job's own exit code:

0 3 * * * curl -fsS -m 10 --retry 3 -o /dev/null https://api.sigtake.com/ping/<token>/start; /path/to/your-job.sh; curl -fsS -m 10 --retry 3 -o /dev/null https://api.sigtake.com/ping/<token>/$?

Every part is there for a reason:

• /start then /$?: the start opens a run, so Sigtake records how long it took and can tell when one hangs. $? is the job's exit code because nothing runs between the job and the curl. 0 is a success; 1 to 255 opens the alert immediately instead of waiting for the deadline.
• ; and not && after the start: if the network blips on the start ping, the job still runs.
• -f makes curl fail on a 404 or 429, so cron mails you the error. -sS stays quiet unless something fails, and -o /dev/null discards the empty 204 response, so a healthy job sends no mail at all.
• -m 10 --retry 3: a slow network can't hang your cron for minutes, and one dropped request doesn't lose the ping.

If you only want the outcome, drop the start ping and keep /path/to/your-job.sh; curl … /ping/<token>/$?. You lose durations and hang detection. A plain request to the URL, with no suffix, counts as a success, and /fail as a failure, for tools that can't pass an exit code. Use if job; then curl …; else curl …/fail; fi rather than job && curl … || curl …/fail: with the second form, a failed success ping would report a job failure that never happened.

Jobs that have no schedule

Not every job runs on a clock. A queue consumer, a webhook handler or a serverless function runs when something triggers it, maybe twice a minute, maybe once a week. There's no "next ping due", so waiting for one would either never alert or alert all the time.

For these, a heartbeat's schedule is On demand. It never alerts on silence. It alerts when a run reports a failure, or when a run passes its maximum duration. That maximum is required here, because it's the only thing that can see a run that died without reporting: a process killed by a timeout or an out-of-memory error never reaches its catch block.

On-demand jobs often overlap, so each run sends its own id with ?rid=, and Sigtake tracks every run separately: its start, its duration, its deadline. One hung run alerts even while others finish fine. The panel shows the wrapper in Shell, curl, Node.js and Python; this is the Node.js one:

// Node 18+. Set SIGTAKE_PING_URL=https://api.sigtake.com/ping/<token>
// One run id per run; never put a secret in it.
import { randomUUID } from 'node:crypto';

const BASE = process.env.SIGTAKE_PING_URL;
const ping = (path, rid) =>
  fetch(`${BASE}${path}?rid=${encodeURIComponent(rid)}`, { signal: AbortSignal.timeout(5000) }).catch(() => {});

export async function monitored(job) {
  const rid = randomUUID();
  await ping('/start', rid);
  try {
    const result = await job();
    await ping('', rid);
    return result;
  } catch (err) {
    await ping('/fail', rid);
    throw err;
  }
}

// await monitored(() => yourJob());

The pings are awaited before the run ends, because a runtime that freezes or exits after your handler returns may never send a request still in flight. A failed ping is swallowed: monitoring must never break the job it watches. The run id isn't a secret and can show up in logs.

One rule keeps a failure from being hidden by a neighbour. After a failure, the incident closes with the first run that started after the failure and finished well. A run that was already in flight when the failure happened proves nothing about the job now, so its success is recorded but doesn't resolve anything.

If you also need to know when an on-demand job stops being triggered at all, add a second heartbeat with a long interval and ping it from the same code, say once a day.

Choosing the grace window

What the grace has to cover depends on when the job pings:

• With /start, the start ping is what meets the schedule, so the grace only absorbs a late start: a busy machine, a slow container boot. A few minutes is usually plenty. How long the run itself may take is the separate Alert if a run takes longer than setting.
• Without /start, the only ping comes at the end, so the grace must cover the job's whole run time plus a margin. Look at the longest run you've seen and add room for a bad day.

The shortest grace, and the shortest maximum duration, is one minute, because Sigtake checks deadlines once a minute: an alert lands within about a minute of its deadline. Match the grace to the schedule. A job that runs every minute isn't watched by a grace of several hours, and a grace of a day on a daily job means you hear about Monday's failure on Tuesday.

Dedicated tools vs. Heartbeats inside your alerting

Heartbeat monitoring isn't new, and the model is the same everywhere: a URL per job, a deadline, an alert on absence. Healthchecks.io, Cronitor and Dead Man's Snitch are mature tools built only for this, and they do it well.

Where Sigtake Heartbeats differs is where the alert goes. It enters the same pipeline as the rest of your alerts: the team that owns the job, that team's Slack, Teams or email channels in the project, the in-app bell, deduplication and automatic resolution. There's no second tool with its own user list and its own notification wiring to keep in sync.

Where it doesn't: Healthchecks.io can attach the job's output to a ping, and Sigtake doesn't read the ping body at all. Cronitor has status pages and on-call features that Sigtake doesn't. If you only need to watch jobs and you already use one of these happily, there's no reason to switch.

What Heartbeats does not do

• It proves the job finished, not that it was right. Ping after you've checked the result.
• No job output. The body of a ping is never read, so there are no logs in Heartbeats; keep them where they are.
• It isn't an on-call system. There are no rotations, escalations, phone calls or SMS. A heartbeat alerts a team.
• A heartbeat without a team reaches nobody. It still goes Down, but nobody is notified.
• Minutes, not seconds. Schedules, grace and maximum durations start at one minute.
• On demand never alerts on silence, by design. Pair it with an interval heartbeat if that matters.
• History is two weeks. Each heartbeat keeps its last 14 days of pings.
• The token lives in the URL, so it can end up in shell history or a repository. Rotate it from the heartbeat's page; you can keep the old URL working for an hour, a day or a week while you update your jobs.

Common questions

Does cron tell you when a job doesn't run?

No. Cron can mail a job's output when it runs, but it has no notion of a job that was expected and didn't happen. Something outside the job has to expect it.

What's the difference between error monitoring and heartbeat monitoring?

Error monitoring reports what a running program throws. Heartbeat monitoring reports what didn't arrive. A job that never starts throws nothing, so only the second one sees it.

How much grace should I give a job?

With a start ping, a few minutes to absorb a late start. Without one, the job's longest run plus a margin. See choosing the grace window.

Is this the same as an uptime check?

They're opposites that complement each other. An uptime check calls your service and watches it answer. A heartbeat waits for your job to call in. A server can answer every uptime check while its nightly backup hasn't run in a month.