Cron job monitoring: how to know when a job fails

A cron job fails quietly because cron has only one task: start a command at a set time. It does not check that the command worked, finished, or did anything useful. If nobody reads the output, a failure looks the same as a success.

These are the nine failure modes worth knowing.

In most rows there is no error message to find, because the code that would have written it never ran or never finished.

That is why checking for errors is not enough. You need to check for the absence of success.

Logs and cron mail are not enough because both depend on the job running and on a person reading the result. Neither one can tell you about a job that never started.

Log files

A log is a record you have to go and read. If the nightly backup stops on Tuesday, the log simply has no new lines. Nothing is sent to you.

A log also lives on the same server as the job. If the disk fills up or the server is gone, the log goes with it.

Cron mail

Cron can email the output of a job to the address in the MAILTO variable, or to the owner of the crontab. This has four weak points:

• It needs a working mail server on the host. Many cloud servers and most containers do not have one, so the output is dropped.
• It sends mail whenever there is output. A healthy but chatty job sends mail every run, and people learn to ignore or filter it.
• It only fires when the job runs. A job that never starts produces no output and no mail.
• A hung job sends nothing. The mail is only sent when the command ends.

Keep your logs. They are how you find out why a job failed. They are not how you find out that it failed.

Heartbeat monitoring means the job tells an outside service "I finished" each time it succeeds, and the service alerts you when that message stops arriving. The alert comes from silence, not from an error.

It works in four steps:

1. You create a heartbeat monitor. It gives you a unique URL.
2. You tell the monitor how often to expect a request, for example once every 24 hours.
3. Your job requests the URL as its last step, and only when it succeeded.
4. If no request arrives within the expected time plus a grace period, the monitor alerts you.

The name "dead man's switch" comes from machines that stop when the operator lets go of a control.

Why it catches every failure in the table

The monitor does not need to know what went wrong. It only knows a success did not arrive.

• Never started, crashed, or hung: no ping is sent.
• Wrong timezone or ran late: the ping arrives outside its window.
• Disk full or broken environment: the command exits with an error, so no ping is sent.

The monitor has to run outside the server it watches. A check that lives on the same machine dies with the machine.

In PULSX, a heartbeat monitor gives you a URL of the form https://pulsx.net/api/beat/<your-token>. If no request arrives within the interval plus the grace period, you get an alert by email, Slack, Discord, webhook or push to the Android app. You can create one for free, with no card.

Set the interval to the gap between two runs, and set the grace period to cover how much the finish time can move from one run to the next. Too little grace gives false alerts. Too much grace delays real ones.

The interval

The interval is the time between two scheduled runs. A job on 0 2 * * * runs once a day, so the interval is 24 hours. A job on */5 * * * * has an interval of 5 minutes.

If you are not sure what a schedule means, paste it into the cron expression tool to see the next run times.

The grace period

The grace period is extra waiting time after the interval has passed. Work it out from three numbers:

1. Runtime spread. Your slowest normal run minus your fastest normal run.
2. Start delay. How late the scheduler may start the job. Plain cron on a healthy server starts within the minute. Hosted schedulers can be much later.
3. Ping retry time. How long your ping command may spend retrying.

Add them up, then round up so that one slow night does not wake you.

Worked example: a nightly backup

The backup starts at 02:00. Over the last month the fastest run took 12 minutes and the slowest took 25 minutes.

• Runtime spread: 25 minus 12 = 13 minutes.
• Start delay: under 1 minute.
• Ping retry time: about 91 seconds at worst with the curl command in the next section. Call it 2 minutes.
• Total: 13 + 1 + 2 = 16 minutes. Round up to 30 minutes.

Now follow one bad night. Monday's run finishes at 02:12 and pings. The monitor expects the next ping by 02:12 on Tuesday. Tuesday's run hangs. At 02:42 (02:12 plus 30 minutes of grace) the alert goes out.

With a 5 minute grace period, a normal slow run that finished at 02:25 would have raised a false alert at 02:17.

For the 5 minute job, a 5 minute grace period means one missed run is forgiven and two in a row raise an alert. That is a sensible default for frequent jobs. It is also the default grace period in PULSX.

Put && and a curl command after your job. The && means curl runs only when the job exits with status 0, which is the Unix signal for success.

# m h dom mon dow  command
0 2 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 5 -o /dev/null https://pulsx.net/api/beat/<your-token>

Each part has a job:

The worst case is 6 attempts of 10 seconds each, plus waits of 1, 2, 4, 8 and 16 seconds. That is 60 + 31 = 91 seconds, which is the retry number used in the grace period sum above.

Make sure the exit status is honest

The && only helps if your script exits with a non-zero status when it fails. A shell script keeps going after a failed command unless you tell it to stop. Start bash scripts with this line:

#!/usr/bin/env bash
set -euo pipefail

-e stops the script at the first failed command. -u treats an unset variable as an error. pipefail matters for backups: without it, pg_dump mydb | gzip > backup.sql.gz reports success when pg_dump fails, because the status of a pipeline is the status of its last command.

Two crontab traps

• The percent sign. In a crontab line, an unescaped % is turned into a newline. A command such as date +%F must be written as date +\%F. The crontab(5) manual page describes this.
• The token is a secret. Anyone who has the URL can send a ping for your job. Keep it out of public repositories.

The rule is the same on every scheduler: run the ping as a final step that only runs after success. Only the syntax changes.

systemd timer

A systemd timer needs two files. The service says what to run, and the timer says when.

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly database backup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 5 -o /dev/null https://pulsx.net/api/beat/<your-token>
TimeoutStartSec=30min

# /etc/systemd/system/backup.timer
[Unit]
Description=Run the backup at 02:00 every night

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

Turn it on with systemctl enable --now backup.timer and check the next run with systemctl list-timers.

• ExecStartPost runs only after ExecStart has succeeded, so it does the job of &&.
• TimeoutStartSec=30min stops a run that is still going after 30 minutes. For a oneshot service there is no time limit unless you set one.
• Persistent=true runs a missed job after the machine boots, if it was off at 02:00.

systemd will not start a second copy of a service that is still running, so overlap is handled for you. The systemd.timer manual lists every option.

Kubernetes CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"
  timeZone: "Europe/London"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 600
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: 1800
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: backup
              image: registry.example.com/backup:1.4.2
              command: ["/bin/sh", "-c"]
              args:
                - /app/backup.sh && curl -fsS -m 10 --retry 5 -o /dev/null "$HEARTBEAT_URL"
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: backup-heartbeat
                      key: url

• concurrencyPolicy: Forbid skips a new run while the previous one is still going.
• startingDeadlineSeconds: 600 lets a late run start up to 10 minutes after its slot, then counts it as missed.
• activeDeadlineSeconds: 1800 ends a run that is still going after 30 minutes.
• backoffLimit: 1 retries a failed run once.

The image must contain curl. Without timeZone, the schedule follows the timezone of the cluster's controller, which is often UTC. The Kubernetes CronJob documentation covers the limits.

GitHub Actions scheduled workflow

name: Nightly export
on:
  schedule:
    - cron: "17 3 * * *"
jobs:
  export:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/export.sh
      - name: Report success
        run: curl -fsS -m 10 --retry 5 -o /dev/null "$HEARTBEAT_URL"
        env:
          HEARTBEAT_URL: ${{ secrets.HEARTBEAT_URL }}

A step runs only if the steps before it succeeded, so the last step is your success ping. Three things are specific to GitHub:

• The schedule is read as UTC.
• Runs can start late when GitHub is busy, and the start of each hour is the busiest time. That is why the example uses minute 17 and not minute 0. Give these jobs a long grace period, such as 60 minutes.
• Schedules can switch off. In a public repository, GitHub disables scheduled workflows after 60 days with no repository activity. A heartbeat alerts you when the runs stop.

Send it as the last line of the program, after the real work has returned without an error. Give the request a timeout so a network problem cannot hang the job.

Python

import sys
import urllib.request

HEARTBEAT_URL = "https://pulsx.net/api/beat/<your-token>"

def main():
    run_export()  # your real work; let it raise on failure

if __name__ == "__main__":
    main()
    try:
        urllib.request.urlopen(HEARTBEAT_URL, timeout=10)
    except Exception as err:
        print(f"heartbeat failed: {err}", file=sys.stderr)

If main() raises an error, Python exits before it reaches the ping. This uses only the standard library, so there is nothing to install.

Node

const HEARTBEAT_URL = process.env.HEARTBEAT_URL;

async function main() {
  await runExport(); // your real work; let it throw on failure
}

main()
  .then(() => fetch(HEARTBEAT_URL, { signal: AbortSignal.timeout(10000) }))
  .catch((err) => {
    console.error(err);
    process.exitCode = 1;
  });

This needs Node 18 or newer, where fetch is built in. AbortSignal.timeout(10000) cancels the request after 10 seconds.

Ping from inside the code, or from the scheduler?

Both work. Pinging from the scheduler line (&& curl) keeps monitoring out of your code and works for any language. Pinging from inside the code lets you add your own checks first.

Those checks are worth adding. A backup that exits with status 0 but wrote a 0 byte file is still a failure. Check the result before you ping, for example that the file is larger than a size you expect, or that the export wrote more than 0 rows.

You catch a hung job by putting a limit on how long it may run. There are two ways to do that: report both the start and the finish, or kill the job after a fixed time.

The general idea: start and finish signals

In this pattern the job sends one signal when it starts and another when it finishes. A monitor that understands both can tell three states apart:

• No start signal: the job never ran.
• Start but no finish within the allowed run time: the job hung or crashed.
• Start and finish: the job worked, and the gap between the two is the run time.

With a single finish ping

PULSX heartbeat monitors use one ping, sent when the job finishes. A hung job is still caught, because the finish ping never arrives. The alert goes out when the interval plus the grace period has passed.

Look at the timing. The monitor counts from the last good ping, which was yesterday's finish. So a backup that hangs at 02:01 today is reported at about 02:42 today, using the numbers from the earlier example. That is close to what a start signal would give you.

Kill the job after a fixed time

Whatever monitor you use, give the job a hard limit so it cannot hold a lock or a database connection for ever. On Linux, the timeout command does this:

0 2 * * * timeout -k 30s 30m /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 5 -o /dev/null https://pulsx.net/api/beat/<your-token>

After 30 minutes, timeout sends the job a stop signal and exits with status 124. Because 124 is not 0, the && skips the ping. The -k 30s part sends a kill signal if the job is still alive 30 seconds later.

Set the limit above your slowest normal run. For a job that takes 12 to 25 minutes, 30 minutes fits.

Wrap the command in flock. It takes a lock on a file before it runs your command, and a second copy that cannot get the lock exits straight away.

*/5 * * * * flock -n /var/lock/sync.lock /usr/local/bin/sync.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://pulsx.net/api/beat/<your-token>

• -n means "do not wait". If another run holds the lock, flock exits with status 1 and your script does not start.
• The lock file can be any path that the job's user is allowed to write to.
• The lock is released when the process ends, even if it crashes. This is the advantage over a hand-made PID file, which can be left behind and block every later run.

Why overlap is dangerous

Say a sync job runs every 5 minutes and normally takes 1 minute. One day the remote API is slow and a run takes 12 minutes. Without a lock, cron starts a second copy at minute 5 and a third at minute 10.

Now three copies compete for the same rows and the same API quota. Each one gets slower, so more copies pile up.

How this fits with the heartbeat

When flock -n skips a run, it exits with status 1, so no ping is sent for that slot. That is correct: no work was done.

The run that holds the lock pings when it finishes. If it never finishes, the pings stop and you get an alert. Add timeout inside the lock to make sure a stuck run is cleared:

*/5 * * * * flock -n /var/lock/sync.lock timeout 4m /usr/local/bin/sync.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://pulsx.net/api/beat/<your-token>

flock only protects one machine. If the same crontab is installed on two servers, both will run the job. Pick one server to own each job, or use a lock in a shared database.

Write every path in full, set the variables the job needs at the top of the crontab, and say which timezone the schedule is in. Cron does not run your command in the same environment as your terminal.

PATH and environment

When you log in, your shell reads profile files that set PATH and other variables. Cron does not read them. It runs commands with /bin/sh and a short PATH, commonly just /usr/bin:/bin.

So a command installed in /usr/local/bin, or by a version manager in your home folder, is "not found" under cron. There are two fixes:

• Use full paths: /usr/local/bin/node, not node. Find the path with command -v node.
• Set PATH on its own line at the top of the crontab.

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=""

0 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1 && curl -fsS -m 10 --retry 5 -o /dev/null https://pulsx.net/api/beat/<your-token>

To test a job the way cron will run it, start it with an empty environment:

env -i SHELL=/bin/sh PATH=/usr/bin:/bin HOME="$HOME" /bin/sh -c '/usr/local/bin/backup.sh'

Timezone

Cron uses the timezone of the server. Many servers are set to UTC. If you write 0 2 * * * meaning 02:00 in London, the job runs at 03:00 London time in summer, when London is one hour ahead of UTC.

Check the server with timedatectl or date. Some cron versions accept a CRON_TZ=Europe/London line to set the timezone for the lines below it. Check your own system's crontab manual before you rely on it.

Clock changes

If the server uses a timezone with summer time, one hour a year does not exist and another happens twice. On a server set to London time, there is no 01:30 on the night the clocks go forward, and there are two on the night they go back. Depending on your cron version, a job in that hour can be skipped or run twice.

The simple fix is to keep servers on UTC, or to keep jobs out of the hour that changes. A heartbeat tells you when a run was skipped. It does not tell you a job ran twice, so jobs must also be safe to run twice.

Disk space

A full disk often produces a file that exists but is empty or cut short. Delete old output as part of the job, and check the size of the new file before you report success.

Go through this list once for each job. It covers every failure in the table at the top of this guide.

The job itself

• The script exits with a non-zero status on any failure (set -euo pipefail in bash).
• Every command uses a full path, or PATH is set in the crontab.
• Every % in the crontab line is escaped as \%.
• Network calls inside the job have their own timeouts.
• The job checks its own result (file size, row count) before it reports success.
• Running the job twice does no harm.

The schedule

• You know which timezone the schedule runs in, and it is written in a comment above the line.
• The job is not in an hour that a clock change skips or repeats.
• The job is installed on one server only, or it holds a shared lock.

The safety net

• flock -n stops overlapping runs.
• timeout (or the scheduler's own limit) ends a hung run.
• The heartbeat ping is chained with &&, has -m and --retry, and runs last.
• The interval matches the schedule, and the grace period comes from real run times.
• The alert goes to a place a person reads, not to a muted channel.

The test

Test the alert once, on purpose. Comment out the crontab line, wait for the interval plus the grace period, and confirm the alert reaches you. Then put the line back.

An alert you have never seen fire is an alert you cannot trust. When a real failure happens, write down what you learned. The incident postmortem guide has a format for that.

Scheduled jobs are only one part of what can break. For websites, APIs and ports, see how to monitor uptime.