What you will learn

Write a watchdog that detects a wedged or crashed service, restarts it with exponential backoff, and gives up after a failure ceiling instead of restart-looping forever and masking a real outage.

Before you begin

Assumes a systemd-managed service, but the health check and backoff logic apply equally to any process you can start/stop/probe. Decide your failure ceiling before deploying — too low pages needlessly, too high hides an incident.

1. Define a real health check, not just "is it running"

#!/usr/bin/env bash
set -Eeuo pipefail

SERVICE="myapp.service"
HEALTH_URL="http://127.0.0.1:8080/healthz"
MAX_ATTEMPTS=5
BASE_DELAY=5

is_healthy() {
    curl -fsS --max-time 3 "$HEALTH_URL" >/dev/null 2>&1
}

A process being alive (systemctl is-active) is not the same as it serving traffic correctly — checking an HTTP health endpoint catches deadlocks a liveness check misses.

2. Restart with exponential backoff and a ceiling

attempt=0
while ! is_healthy; do
    attempt=$((attempt + 1))
    if (( attempt > MAX_ATTEMPTS )); then
        echo "giving up after ${MAX_ATTEMPTS} attempts" >&2
        exit 1
    fi

    delay=$(( BASE_DELAY * (2 ** (attempt - 1)) ))
    echo "unhealthy, restart attempt ${attempt}/${MAX_ATTEMPTS} (waiting ${delay}s first)"
    sleep "$delay"

    systemctl restart "$SERVICE"
    sleep 3   # give it a moment before the next health probe
done

echo "healthy after ${attempt} restart(s)"

Backoff (5s, 10s, 20s, 40s, 80s) avoids hammering a service that needs time to recover — e.g. reconnecting to a database — while still recovering fast from a one-off blip.

3. Wire the ceiling into an alert, not a silent exit

trap 'if [[ $? -ne 0 ]]; then ./webhook-alert-dispatcher.sh "watchdog gave up on ${SERVICE}"; fi' EXIT

Example output

unhealthy, restart attempt 1/5 (waiting 5s first)
unhealthy, restart attempt 2/5 (waiting 10s first)
healthy after 2 restart(s)

Verify the result

Kill the service's health endpoint manually (e.g. block its port with iptables) and confirm the watchdog restarts it, then confirm it gives up and alerts after MAX_ATTEMPTS if the service stays broken.

Troubleshooting

If the watchdog restarts endlessly without ever hitting the ceiling, check that attempt is scoped to one watchdog invocation, not reset by an external timer calling the script fresh every minute — put the whole loop under a single flock if it runs from cron.

Next steps

Combine with pidfile-heartbeat-supervisor for services with no HTTP endpoint, and flock-cron-concurrency-guard so the watchdog itself can't run twice concurrently.