Category: DevOps (page 1 of 1)

I once tested a database migration by running it directly against production during a quiet hour, because there was nowhere else to run it. It worked – that time. The next similarly “safe” migration locked a table for four minutes during a traffic spike we hadn’t anticipated, because the quiet-hour assumption didn’t hold and there was no environment where that kind of mistake was cheap. That’s the entire case for staging: it’s not about mirroring production perfectly, it’s about having somewhere for things to go wrong that isn’t in front of real users.

What staging is actually for

A staging environment earns its keep on a specific, narrow set of problems: verifying a database migration before it touches real data, confirming a third-party integration still works against its real API (not a mock) with the actual credentials and rate limits you’ll hit in production, and catching environment-specific bugs that never show up on a laptop – path casing issues, missing environment variables, a build step that behaves differently without dev tooling installed.

It’s also the only honest place to test anything involving webhooks or callbacks from an external service – a CI provider notifying a deploy hook, a third-party auth provider redirecting back after login, a mail service confirming delivery. These require a publicly reachable URL and real request/response round trips that a local tunnel only approximates. I’ve seen integrations that worked perfectly against a mocked webhook payload in tests fail immediately in production because the real payload had an extra field or a different content type – the kind of mismatch staging exists to catch before it reaches a customer.

It doesn’t need to be a perfect replica. The Twelve-Factor App’s dev/prod parity principle argues for minimizing gaps between environments, but “minimizing” is the operative word – staging needs to use the same backing services and the same deploy process as production, not the same hardware scale or the same customer data volume.

The cheap version

You don’t need a second production-sized cluster. For most small-to-mid projects, a single modest VPS running the same Docker Compose stack as production, behind a subdomain like staging.yourapp.com, covers the cases above. The same docker-compose.yml that defines your production services, pointed at a smaller Postgres instance and a staging-only set of secrets, gets you 90% of the value at a fraction of the infrastructure.

# docker-compose.staging.yml
services:
  app:
    image: myapp:${TAG}
    env_file: .env.staging
    ports:
      - "3000:3000"
  db:
    image: postgres:16
    volumes:
      - staging_db_data:/var/lib/postgresql/data
volumes:
  staging_db_data:

A single small droplet or EC2 instance running this is usually a few dollars a month, and it’s the same deploy artifact (same Docker image) you’ll ship to production, which is the part that actually matters – you’re testing the thing you’ll deploy, not a hand-built approximation of it.

Data without the liability

Copying a real production database dump into staging is tempting and, for anything handling personal data, usually a bad idea – now you have customer data sitting on a lower-security box with looser access controls. Two workable alternatives: a scrubbing script that copies the schema and replaces sensitive columns with generated values (names, emails, anything identifying) before loading it into staging, or a seed script that generates realistic synthetic fixtures from scratch. The seed script is more upfront work but it’s also reusable for local development and for tests, so it tends to pay for itself.

Wiring it into your deploy flow

Staging is only useful if code actually lands there before production, which means it should be automatic, not a manual step someone forgets. A simple CI job that builds the image and deploys it to staging on every merge to your main integration branch keeps it honest:

on:
  push:
    branches: [develop]
jobs:
  deploy-staging:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t myapp:${{ github.sha }} .
      - run: ssh staging-host "docker compose pull && docker compose up -d"

Once this exists, “check it on staging first” stops being an optional courtesy and becomes the default path something takes before reaching real users.

It’s worth protecting staging from becoming a second, unofficial production – the moment someone starts pointing a real client demo or an important internal dashboard at it, people stop feeling free to break it, and the whole point erodes. A basic HTTP auth prompt in front of the staging subdomain, or an IP allowlist if your team works from a known set of networks, keeps it clearly marked as internal without adding real infrastructure.

What to skip

Resist the urge to gold-plate staging. It doesn’t need autoscaling, high availability, or a CDN in front of it – those are production concerns, and building them twice is wasted effort. It needs to run the same code, talk to the same kinds of services, and be disposable enough that you’re not afraid to break it. If staging becomes precious – if people are scared to experiment on it – it’s stopped doing its job, which is to be the place where mistakes are cheap.

Every project I’ve worked on ends up with a .env file within the first week. Then a .env.local. Then someone adds a .env.staging and forgets to tell anyone which variables changed between it and production. Six months in, nobody is entirely sure which keys are actually still used, which ones are stale, and which ones would break the app if rotated. None of this is inevitable – it’s just what happens when secrets management is treated as an afterthought instead of a small, deliberate practice.

The baseline everyone should already have

Before anything more sophisticated: .env belongs in .gitignore, full stop. What belongs in the repo is a .env.example with every key present and placeholder or dummy values filled in – not because it’s a nice-to-have, but because it’s the only reliable documentation of what config a fresh checkout actually needs. If a new variable is added to the app, the pull request that introduces it should also update .env.example. This one habit prevents the most common onboarding failure: “it works on my machine” turning out to mean “I have three environment variables set that nobody told you about.”

The Twelve-Factor App’s config principle is still the clearest framing of why this matters: config that varies between environments (deploy-specific values, credentials) should live outside the codebase entirely, not in checked-in config files with per-environment sections.

Where things actually go wrong

The baseline above prevents accidental commits, but the real failures I see are process failures. A production API key gets pasted into a Slack thread to unblock someone quickly, and now it lives in Slack’s search index forever. A secret gets rotated in production but someone forgets staging, and staging silently starts failing auth calls a week later when a cache expires. Or the classic: a .env file does get committed, months pass, and by the time anyone notices, the key has been indexed by every automated scanner that crawls public GitHub repositories looking for exactly this.

None of these are solved by “be more careful.” They’re solved by removing the manual step where a human has to remember to be careful.

Separate config from secrets

Not everything in a .env file is equally sensitive. A feature flag, an API base URL, a log level – these are configuration, and it’s fine for them to sit in a plain file. A database password, a signing key, a third-party API secret – these need actual secret management: encrypted at rest, access-controlled, and auditable. Lumping both categories into one flat file makes it hard to reason about what’s actually sensitive.

For the second category, a dedicated secrets manager pays for itself quickly, whether that’s a cloud-native option like AWS Secrets Manager, a self-hosted HashiCorp Vault instance, or a lighter tool like Doppler or 1Password’s CLI integration. The specific tool matters less than the properties it gives you: centralized access control, an audit log of who read what and when, and the ability to rotate a value in one place instead of hunting through every environment’s config.

The OWASP Secrets Management Cheat Sheet is worth reading even if you don’t adopt every recommendation in it – the section on avoiding secrets baked into build artifacts is the one I see skipped most often. Building an API key into a Docker image at build time means that key lives in every layer of every image you ever push, including old ones sitting in a registry nobody’s looked at in a year. The safer pattern is injecting secrets at container start, through environment variables or a mounted file the entrypoint reads, so the value never touches the image itself.

Rotating without breaking things

Rotation is where most ad hoc setups fall apart, because swapping a credential atomically usually isn’t possible – there’s a window where the old value is invalid and the new one hasn’t propagated everywhere yet. The reliable pattern is dual credentials: create the new secret alongside the old one, deploy the new value everywhere it’s needed, confirm everything is using it, then revoke the old one. It’s more steps than editing one file, but it’s the difference between a rotation that’s invisible to users and one that causes an outage at 2am because a background worker still had the old database password cached.

Catching leaks before they ship

Even with good habits, mistakes happen – someone hardcodes a key while debugging and forgets to remove it before committing. A pre-commit hook running a secret scanner like gitleaks catches this before it ever reaches a shared branch, scanning the diff for patterns that look like API keys, private keys, or connection strings:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.4
    hooks:
      - id: gitleaks

It takes a few minutes to wire into a repo’s pre-commit config, and it has saved me from at least two accidental commits that would have otherwise required rotating a production credential under time pressure. The same scanner running again in CI, against the full history rather than just the current diff, is cheap insurance against the hook being skipped locally with --no-verify.

None of this is complicated in isolation. The hard part is doing it consistently across a team and across the lifetime of a project, which is exactly why it’s worth automating the boring parts – the example file check, the leak scan, the rotation runbook – rather than relying on everyone remembering the rules every time.

A cron job is easy to set up and easy to forget about. That’s exactly the problem. I’ve lost count of how many times a scheduled task quietly stopped working – a script started exiting early, an API it depended on changed its response shape, a disk filled up – and nobody noticed until someone asked why a report hadn’t updated in three weeks. Cron doesn’t care whether your job actually did its job. It only cares whether the process started.

Why cron failures go unnoticed

By default, cron mails job output to the crontab owner if MAILTO is set and a mail transfer agent is configured on the box. On most modern servers, neither of those things is true. No mail server, no MAILTO, so stdout and stderr just vanish. Even when mail is wired up, a script that catches its own exceptions and exits with status 0 won’t trigger anything – cron only knows the process ran, not whether the work inside it succeeded.

This is the core issue: cron monitors process execution, not job success. A script that fails halfway through, skips its actual task because an API returned an empty response, or silently no-ops because a config file is missing will look identical to a successful run from cron’s point of view.

There’s a second, quieter failure mode: overlapping runs. If a job normally finishes in two minutes but occasionally takes twenty because of a slow upstream API, and cron fires it again five minutes later regardless, you now have two instances writing to the same output file or fighting over the same database rows. Nothing crashes, but the result is subtly wrong. Wrapping the job command in flock prevents this cheaply:

*/5 * * * * flock -n /tmp/myjob.lock /usr/local/bin/myjob.sh

The -n flag makes it non-blocking – if a previous run is still holding the lock, the new invocation exits immediately instead of queuing up, which is almost always what you want for a job that’s supposed to run on a fixed schedule rather than back-to-back.

The dead man’s switch pattern

The fix that actually works is inverting the check: instead of waiting to be told something failed, expect to be told something succeeded, and alert when that confirmation doesn’t show up. This is usually called a dead man’s switch or heartbeat monitor. The job pings an external endpoint at the end of a successful run; a monitoring service watches for that ping and fires an alert if it doesn’t arrive within the expected window.

In practice this is one line added to the end of a script:

#!/bin/bash
set -euo pipefail

/usr/local/bin/generate-daily-report.sh

curl -fsS -m 10 --retry 3 https://hc-ping.com/your-check-uuid

If the report script exits non-zero, set -e stops the script before the curl line ever runs, so no ping means no success. Services like Healthchecks.io let you define the expected schedule (say, “once a day, grace period of one hour”) and will email, Slack, or page you the moment a run doesn’t check in on time. You can self-host equivalents too, but the value is in the pattern, not the specific vendor.

Structured logging and exit codes

Heartbeat pings tell you a job didn’t run cleanly, not why. Pair them with disciplined scripting: set -euo pipefail at the top of every bash script so an unhandled error actually stops execution instead of continuing on garbage state, explicit exit codes for different failure modes, and timestamped log lines so you can reconstruct what happened after the fact.

log() { echo "$(date -Is) $*" >> /var/log/myjob.log; }

log "starting export"
if ! pg_dump mydb > /tmp/export.sql; then
  log "pg_dump failed with exit $?"
  exit 1
fi
log "export finished, $(wc -l < /tmp/export.sql) lines"

This costs almost nothing to add and turns "the job silently didn't work" into "the job failed at the pg_dump step, here's the timestamp, here's the log line right before it."

A lighter self-hosted option

If you're already running Prometheus, you don't need an external SaaS for this. The node_exporter textfile collector lets any script write a metrics file that node_exporter picks up and exposes alongside normal host metrics. A cron job writes its last-success timestamp as a gauge, and an alerting rule fires if that timestamp gets too old:

echo "job_last_success_timestamp $(date +%s)" > /var/lib/node_exporter/textfile_collector/myjob.prom.$$
mv /var/lib/node_exporter/textfile_collector/myjob.prom.$$ /var/lib/node_exporter/textfile_collector/myjob.prom

The atomic rename avoids node_exporter reading a half-written file. From there, a standard Prometheus alert rule like time() - job_last_success_timestamp > 90000 covers you without adding another external dependency to your stack.

What I actually run now

For most projects I don't need Prometheus-grade infrastructure, so the practical setup is a small shell wrapper every cron job calls through, instead of calling the underlying script directly:

#!/bin/bash
set -euo pipefail
START=$(date +%s)
CMD="$1"
CHECK_URL="$2"

if $CMD; then
  DURATION=$(( $(date +%s) - START ))
  echo "$(date -Is) ok ($CMD, ${DURATION}s)" >> /var/log/cron-wrapper.log
  curl -fsS -m 10 "$CHECK_URL" > /dev/null
else
  echo "$(date -Is) FAILED ($CMD)" >> /var/log/cron-wrapper.log
fi

The crontab entry just becomes flock -n /tmp/myjob.lock /usr/local/bin/cron-wrapper.sh /usr/local/bin/myjob.sh https://hc-ping.com/uuid. It's maybe thirty lines of bash, reused across every scheduled job on a server, and it's the difference between finding out a job broke from a dashboard versus finding out from a customer.

None of this is exotic. It's the unglamorous work of treating scheduled jobs with the same seriousness as request-handling code - because a broken cron job that runs "successfully" every night for a month does just as much damage as an outage, just more quietly.