A backup you haven’t restored is a hypothesis, not a backup. I picked up that habit of thinking the hard way, helping a friend recover a server after a bad update wiped its database – the nightly dump job had been running successfully for months, green checkmarks the whole time, and the archive turned out to be truncated because the disk had quietly filled up mid-dump weeks earlier and nobody had checked. The job “succeeded” every night. The data it produced was garbage.

Having copies isn’t the hard part

The classic 3-2-1 rule – three copies of your data, on two different types of media, with one copy offsite – is a reasonable baseline and worth following: a local snapshot plus a copy on a different provider or region protects against both hardware failure and a single provider’s outage taking out your only backup along with your production data. But 3-2-1 answers “where do copies live,” not “will this copy actually work when I need it,” and that second question is where most backup strategies quietly fail.

Automate the restore test, not just the backup

The only way to know a backup is good is to restore it somewhere and check the result, and the only way that happens reliably is if it’s automated rather than left as a task someone means to get around to. A simple version: a scheduled job spins up a throwaway database container, restores the latest backup into it, and runs a handful of sanity checks – row counts on key tables, a checksum on a known-stable table, maybe a smoke-test query that exercises a foreign key relationship.

#!/bin/bash
set -euo pipefail

docker run -d --name restore-test -e POSTGRES_PASSWORD=test postgres:16
sleep 5
gunzip -c /backups/latest.sql.gz | docker exec -i restore-test psql -U postgres

ROWS=$(docker exec restore-test psql -U postgres -tAc "SELECT count(*) FROM orders;")
if [ "$ROWS" -lt 1000 ]; then
  echo "restore test failed: only $ROWS rows in orders" >&2
  exit 1
fi

docker rm -f restore-test

This doesn’t prove the backup is perfect, but it catches the failure mode that actually happens most often: a truncated file, a dump that finished with an error nobody looked at, a schema mismatch after an unrelated migration. Run it weekly and alert on failure the same way you’d alert on any other job, ideally through a heartbeat-style check rather than trusting the backup job’s own exit code.

Application-consistent vs crash-consistent

Filesystem or volume snapshots are convenient because they’re fast and require no application awareness, but a snapshot taken mid-write on a database is only crash-consistent – equivalent to yanking the power cord. Most databases can recover from that using their write-ahead log, but it’s a worse starting position than a clean, application-consistent dump taken with the database’s own tooling, which knows how to produce a coherent point-in-time export. For anything running Postgres or MySQL, pg_dump or a properly configured mysqldump/mariabackup run should be the backup of record, with filesystem snapshots as a fast secondary layer for full-server disaster recovery rather than the only line of defense.

Retention and verification

Keeping every backup forever is wasteful and keeping too few is dangerous – a common pattern is daily backups retained for a couple of weeks, weekly ones for a couple of months, and monthly ones for a year, giving you both a short recovery window for “I need yesterday’s data” and a longer one for “we didn’t notice this was wrong for six weeks.” Whatever the schedule, store a checksum alongside each archive and verify it before trusting the file, since silent corruption in cloud storage is rare but not impossible, and it’s a cheap check to skip only in hindsight:

sha256sum backup-2026-07-27.sql.gz > backup-2026-07-27.sql.gz.sha256
# later, before restoring:
sha256sum -c backup-2026-07-27.sql.gz.sha256

It’s two extra commands in a script that already runs unattended, and it turns “the archive looks the right size” into an actual guarantee that the bytes haven’t changed since the day they were written.

The retention window also has to account for how long a bad change can go unnoticed. If corrupted or incorrectly deleted data sometimes isn’t spotted for a month, a two-week retention policy has already lost the clean copy by the time anyone realizes something needs restoring. This is the actual argument for monthly archives kept for a year rather than aesthetic thoroughness – they’re there for the slow-burning mistakes, not the fast ones.

Tooling that makes this easier

For file and volume-level backups, restic handles encryption, deduplication, and multiple storage backends (S3, Backblaze, SFTP) out of the box, and its restic check command specifically verifies repository integrity – worth running on a schedule alongside your own restore test. Database-native dumps still belong alongside it for anything relational; the two approaches cover different failure modes and neither fully substitutes for the other.

None of this needs to be elaborate. It needs to be automatic, and it needs to actually run the restore, because the backup job that’s never been tested is the one that fails you exactly when you can’t afford it.