What you will learn

Push structured, rate-limited failure alerts from any script to a chat webhook (Slack-compatible), so a batch of failing jobs sends one useful summary instead of flooding the channel.

Before you begin

Store the webhook URL in an environment variable or secret file, never hardcoded in a script checked into version control. Slack-compatible webhooks accept the same JSON payload shape used here.

1. A minimal, correct alert function

#!/usr/bin/env bash
set -Eeuo pipefail

WEBHOOK_URL="${ALERT_WEBHOOK_URL:?ALERT_WEBHOOK_URL not set}"

send_alert() {
    local text="$1"
    local payload
    payload="$(jq -nc --arg text "$text" '{text: $text}')"

    curl -fsS --max-time 5 \
        -H 'Content-Type: application/json' \
        -d "$payload" \
        "$WEBHOOK_URL" >/dev/null
}

Building the payload with jq -nc --arg (rather than a heredoc with variable interpolation) avoids breaking the JSON when $text itself contains a quote or newline.

2. Rate-limit so a flapping check doesn't spam the channel

STATE_FILE="/var/tmp/alert-last-sent"
MIN_INTERVAL_SECONDS=300

send_alert_throttled() {
    local text="$1" now last=0
    now="$(date +%s)"
    [[ -f "$STATE_FILE" ]] && last="$(cat "$STATE_FILE")"

    if (( now - last < MIN_INTERVAL_SECONDS )); then
        echo "alert suppressed (rate limit): $text" >&2
        return 0
    fi

    send_alert "$text"
    echo "$now" > "$STATE_FILE"
}

A file-based cooldown is enough for a single host; for a fleet of hosts alerting on the same condition, key the state file by check name so unrelated alerts don't suppress each other.

3. Batch multiple failures into one message

declare -a FAILURES=()

record_failure() { FAILURES+=("$1"); }

flush_failures() {
    (( ${#FAILURES[@]} == 0 )) && return 0

    local body="*${#FAILURES[@]} failure(s) on $(hostname):*"$'\n'
    for f in "${FAILURES[@]}"; do
        body+="- ${f}"$'\n'
    done

    send_alert_throttled "$body"
    FAILURES=()
}

trap flush_failures EXIT

Collecting failures into an array and flushing once at exit turns "five separate pings" into one readable digest — call record_failure wherever a check fails during the script, and let the EXIT trap send the summary.

Example output

alert suppressed (rate limit): *1 failure(s) on db01:*
- disk usage at 91%

Slack message body:

*2 failure(s) on db01:*
- backup verification failed
- certificate expires in 3 days

Verify the result

Point WEBHOOK_URL at a test channel, trigger two failures in one run, and confirm exactly one message arrives containing both. Trigger a third run within the cooldown window and confirm it's suppressed with a log line, not silently dropped.

Troubleshooting

If curl returns 400, print $payload before sending — the most common cause is an unescaped control character that slipped past jq because it was injected via printf instead of --arg.

Next steps

Call record_failure from disk-space-cleanup-guard, tls-cert-expiry-checker, and backup-restore-verification-job to centralize all failure notifications through one throttled path.