What you will learn

Write a reusable retry() function that wraps any command with exponential backoff and jitter, so a flaky network call fails gracefully instead of aborting the whole pipeline on the first blip.

Before you begin

This pattern works for any command, not just network calls — anything with transient failures (lock contention, rate limits, cold caches) benefits. Jitter matters once more than one caller retries against the same backend simultaneously.

1. Write the retry function

#!/usr/bin/env bash
set -Eeuo pipefail

retry() {
    local max_attempts="$1" base_delay="$2"
    shift 2
    local attempt=1

    until "$@"; do
        if (( attempt >= max_attempts )); then
            echo "retry: giving up after ${attempt} attempts: $*" >&2
            return 1
        fi

        local jitter=$(( RANDOM % 1000 ))                       # 0-999 ms
        local delay_ms=$(( base_delay * (2 ** (attempt - 1)) * 1000 + jitter ))
        echo "retry: attempt ${attempt} failed, waiting ${delay_ms}ms" >&2

        sleep "$(awk "BEGIN { printf \"%.3f\", ${delay_ms}/1000 }")"
        attempt=$((attempt + 1))
    done
}

"$@" after shift 2 forwards the wrapped command and its own arguments untouched — retry never has to know or care what it's retrying.

2. Use it against a real flaky call

fetch_manifest() {
    curl -fsS --max-time 5 "https://api.example.com/manifest.json" -o /tmp/manifest.json
}

if retry 5 1 fetch_manifest; then
    echo "manifest fetched"
else
    echo "manifest fetch failed permanently" >&2
    exit 1
fi

3. Distinguish retryable from fatal failures

fetch_manifest_smart() {
    local http_code
    http_code="$(curl -fsS -o /tmp/manifest.json -w '%{http_code}' \
                 --max-time 5 "https://api.example.com/manifest.json" || echo "000")"

    case "$http_code" in
        200) return 0 ;;
        429|502|503|504) return 1 ;;   # retryable
        *) echo "fatal http ${http_code}, not retrying" >&2; exit 2 ;;
    esac
}

Retrying a 404 or 400 five times just delays an inevitable failure — only backoff on codes that mean "try again later."

Example output

retry: attempt 1 failed, waiting 1247ms
retry: attempt 2 failed, waiting 2189ms
manifest fetched

Verify the result

Point fetch_manifest at an unreachable host and confirm it gives up after exactly max_attempts with a non-zero exit code, not a hang.

Troubleshooting

If delays look identical across runs, RANDOM wasn't reseeded — each subshell gets a fresh seed from the OS by default, so this only bites if you've explicitly set RANDOM=<fixed> earlier in the script.

Next steps

Layer this under ssh-fleet-command-executor for per-host retries, or under webhook-alert-dispatcher so a transient alerting-service outage doesn't drop an alert.