A cron job is easy to set up and easy to forget about. That’s exactly the problem. I’ve lost count of how many times a scheduled task quietly stopped working – a script started exiting early, an API it depended on changed its response shape, a disk filled up – and nobody noticed until someone asked why a report hadn’t updated in three weeks. Cron doesn’t care whether your job actually did its job. It only cares whether the process started.

Why cron failures go unnoticed

By default, cron mails job output to the crontab owner if MAILTO is set and a mail transfer agent is configured on the box. On most modern servers, neither of those things is true. No mail server, no MAILTO, so stdout and stderr just vanish. Even when mail is wired up, a script that catches its own exceptions and exits with status 0 won’t trigger anything – cron only knows the process ran, not whether the work inside it succeeded.

This is the core issue: cron monitors process execution, not job success. A script that fails halfway through, skips its actual task because an API returned an empty response, or silently no-ops because a config file is missing will look identical to a successful run from cron’s point of view.

There’s a second, quieter failure mode: overlapping runs. If a job normally finishes in two minutes but occasionally takes twenty because of a slow upstream API, and cron fires it again five minutes later regardless, you now have two instances writing to the same output file or fighting over the same database rows. Nothing crashes, but the result is subtly wrong. Wrapping the job command in flock prevents this cheaply:

*/5 * * * * flock -n /tmp/myjob.lock /usr/local/bin/myjob.sh

The -n flag makes it non-blocking – if a previous run is still holding the lock, the new invocation exits immediately instead of queuing up, which is almost always what you want for a job that’s supposed to run on a fixed schedule rather than back-to-back.

The dead man’s switch pattern

The fix that actually works is inverting the check: instead of waiting to be told something failed, expect to be told something succeeded, and alert when that confirmation doesn’t show up. This is usually called a dead man’s switch or heartbeat monitor. The job pings an external endpoint at the end of a successful run; a monitoring service watches for that ping and fires an alert if it doesn’t arrive within the expected window.

In practice this is one line added to the end of a script:

#!/bin/bash
set -euo pipefail

/usr/local/bin/generate-daily-report.sh

curl -fsS -m 10 --retry 3 https://hc-ping.com/your-check-uuid

If the report script exits non-zero, set -e stops the script before the curl line ever runs, so no ping means no success. Services like Healthchecks.io let you define the expected schedule (say, “once a day, grace period of one hour”) and will email, Slack, or page you the moment a run doesn’t check in on time. You can self-host equivalents too, but the value is in the pattern, not the specific vendor.

Structured logging and exit codes

Heartbeat pings tell you a job didn’t run cleanly, not why. Pair them with disciplined scripting: set -euo pipefail at the top of every bash script so an unhandled error actually stops execution instead of continuing on garbage state, explicit exit codes for different failure modes, and timestamped log lines so you can reconstruct what happened after the fact.

log() { echo "$(date -Is) $*" >> /var/log/myjob.log; }

log "starting export"
if ! pg_dump mydb > /tmp/export.sql; then
  log "pg_dump failed with exit $?"
  exit 1
fi
log "export finished, $(wc -l < /tmp/export.sql) lines"

This costs almost nothing to add and turns "the job silently didn't work" into "the job failed at the pg_dump step, here's the timestamp, here's the log line right before it."

A lighter self-hosted option

If you're already running Prometheus, you don't need an external SaaS for this. The node_exporter textfile collector lets any script write a metrics file that node_exporter picks up and exposes alongside normal host metrics. A cron job writes its last-success timestamp as a gauge, and an alerting rule fires if that timestamp gets too old:

echo "job_last_success_timestamp $(date +%s)" > /var/lib/node_exporter/textfile_collector/myjob.prom.$$
mv /var/lib/node_exporter/textfile_collector/myjob.prom.$$ /var/lib/node_exporter/textfile_collector/myjob.prom

The atomic rename avoids node_exporter reading a half-written file. From there, a standard Prometheus alert rule like time() - job_last_success_timestamp > 90000 covers you without adding another external dependency to your stack.

What I actually run now

For most projects I don't need Prometheus-grade infrastructure, so the practical setup is a small shell wrapper every cron job calls through, instead of calling the underlying script directly:

#!/bin/bash
set -euo pipefail
START=$(date +%s)
CMD="$1"
CHECK_URL="$2"

if $CMD; then
  DURATION=$(( $(date +%s) - START ))
  echo "$(date -Is) ok ($CMD, ${DURATION}s)" >> /var/log/cron-wrapper.log
  curl -fsS -m 10 "$CHECK_URL" > /dev/null
else
  echo "$(date -Is) FAILED ($CMD)" >> /var/log/cron-wrapper.log
fi

The crontab entry just becomes flock -n /tmp/myjob.lock /usr/local/bin/cron-wrapper.sh /usr/local/bin/myjob.sh https://hc-ping.com/uuid. It's maybe thirty lines of bash, reused across every scheduled job on a server, and it's the difference between finding out a job broke from a dashboard versus finding out from a customer.

None of this is exotic. It's the unglamorous work of treating scheduled jobs with the same seriousness as request-handling code - because a broken cron job that runs "successfully" every night for a month does just as much damage as an outage, just more quietly.