Supervise a Process with a PID File and Heartbeat

What you will learn

Track a background job's PID safely and detect a genuinely hung process via a heartbeat file, going beyond a bare kill -0 check that only proves a PID exists — not that it's making progress.

Before you begin

A stale PID file (left behind after a crash) is the classic failure mode this design defends against: reusing a PID that the OS has since assigned to an unrelated process.

1. Write the PID file defensively at startup

#!/usr/bin/env bash
set -Eeuo pipefail

PID_FILE="/var/run/myjob.pid"
HEARTBEAT_FILE="/var/run/myjob.heartbeat"
SELF_NAME="myjob"

if [[ -f "$PID_FILE" ]]; then
    old_pid="$(cat "$PID_FILE")"
    if kill -0 "$old_pid" 2>/dev/null && grep -q "$SELF_NAME" "/proc/${old_pid}/comm" 2>/dev/null; then
        echo "already running as pid ${old_pid}" >&2
        exit 1
    fi
    echo "stale pid file (pid ${old_pid} not running or not ours), removing" >&2
    rm -f "$PID_FILE" "$HEARTBEAT_FILE"
fi

echo $$ > "$PID_FILE"
trap 'rm -f "$PID_FILE" "$HEARTBEAT_FILE"' EXIT

Checking /proc/<pid>/comm in addition to kill -0 guards against the specific case where the PID in a stale file has since been recycled by the OS for an unrelated process — kill -0 alone would wrongly report "already running."

2. Touch a heartbeat from inside the work loop

main_loop() {
    while true; do
        do_unit_of_work
        touch "$HEARTBEAT_FILE"
        sleep 5
    done
}

do_unit_of_work() {
    : # real work goes here
}

main_loop &
wait

3. A separate checker distinguishes "alive" from "making progress"

#!/usr/bin/env bash
# heartbeat-check.sh — run from cron independently of the job itself
set -Eeuo pipefail

PID_FILE="/var/run/myjob.pid"
HEARTBEAT_FILE="/var/run/myjob.heartbeat"
STALE_AFTER_SECONDS=60

[[ -f "$PID_FILE" ]] || { echo "not running (no pid file)"; exit 1; }
pid="$(cat "$PID_FILE")"
kill -0 "$pid" 2>/dev/null || { echo "not running (pid ${pid} dead)"; exit 1; }

[[ -f "$HEARTBEAT_FILE" ]] || { echo "no heartbeat yet"; exit 1; }
last="$(stat -c %Y "$HEARTBEAT_FILE")"
now="$(date +%s)"

if (( now - last > STALE_AFTER_SECONDS )); then
    echo "HUNG: pid ${pid} alive but heartbeat stale by $((now - last))s" >&2
    exit 2
fi

echo "healthy: pid ${pid}, heartbeat ${STALE_AFTER_SECONDS}s ago or fresher"

This is the piece a plain PID check misses: a process can be alive (passing kill -0) while deadlocked on a lock or stuck in an infinite loop that never reaches the touch. The separate checker catches that from outside the process.

Example output

healthy: pid 4021, heartbeat 60s or fresher

hung case:

HUNG: pid 4021 alive but heartbeat stale by 143s

Verify the result

Send STOP (not KILL) to the job's PID with kill -STOP <pid> to freeze it without killing it, then run the checker and confirm it reports HUNG once the heartbeat goes stale — kill -0 alone would still report it as running.

Troubleshooting

If the checker always reports "not running" right after start, confirm the heartbeat file's directory survives a reboot (/var/run is often tmpfs and cleared) — use a persistent path if the check must survive a host restart before the job re-registers.

Next steps

Wire HUNG/exit code 2 from the checker into auto-heal-service-watchdog's restart logic, and into webhook-alert-dispatcher for paging.