What you will learn

Turn a night's raw application logs into a ranked report of new or spiking error signatures using awk and sort, so a human reviews a ten-line summary instead of grepping megabytes of text every morning.

Before you begin

This assumes semi-structured log lines with a level field (ERROR, WARN, etc.); adjust the field extraction to match your actual log format. Keep yesterday's signature counts around for comparison — that's what turns a static top-N list into an anomaly report.

1. Normalize error lines into comparable signatures

#!/usr/bin/env bash
set -Eeuo pipefail

LOG_FILE="/var/log/myapp/app.log"
STATE_DIR="/var/lib/myapp/log-signatures"
mkdir -p "$STATE_DIR"

TODAY="$(date -u +%F)"
YESTERDAY="$(date -u -d 'yesterday' +%F)"
TODAY_COUNTS="${STATE_DIR}/${TODAY}.counts"

# Strip timestamps, request IDs, and numbers so similar errors collapse into one signature.
grep ' ERROR ' "$LOG_FILE" \
  | sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:.]+Z//; s/req-[a-f0-9-]+/req-<id>/; s/[0-9]+/<N>/g' \
  | sort | uniq -c | sort -rn > "$TODAY_COUNTS"

Replacing timestamps, request IDs, and raw numbers with placeholders before counting is the key step — without it, every error line looks unique because it carries a distinct timestamp, and no pattern ever accumulates a count above one.

2. Compare against yesterday to find what's new or spiking

YESTERDAY_COUNTS="${STATE_DIR}/${YESTERDAY}.counts"

awk '
    NR==FNR { yc[$0]=$1; next }
    {
        sig = $0; sub(/^ *[0-9]+ /, "", sig)
        today_n = $1
        yest_n = 0
        for (k in yc) {
            ksig = k; sub(/^ *[0-9]+ /, "", ksig)
            if (ksig == sig) { yest_n = yc[k]; break }
        }
        delta = today_n - yest_n
        if (delta > 5 || yest_n == 0)
            printf "%-6d (was %-4d, +%-4d)  %s\n", today_n, yest_n, delta, sig
    }
' "$YESTERDAY_COUNTS" "$TODAY_COUNTS" 2>/dev/null | sort -rn > "${STATE_DIR}/${TODAY}.anomalies" || \
    cp "$TODAY_COUNTS" "${STATE_DIR}/${TODAY}.anomalies"   # first run: no baseline yet

Flagging both brand-new signatures (yest_n == 0) and ones that jumped by more than a fixed delta catches two different failure shapes: a new bug appearing, and an existing rare error suddenly spiking.

3. Render the top of the report

echo "=== Log anomaly report for ${TODAY} ==="
head -10 "${STATE_DIR}/${TODAY}.anomalies"
echo "(full detail: ${STATE_DIR}/${TODAY}.anomalies)"

# Keep 30 days of history, prune older.
find "$STATE_DIR" -name '*.counts' -mtime +30 -delete

Example output

=== Log anomaly report for 2026-09-19 ===
142    (was 3   , +139)  connection pool exhausted acquiring db01.internal
38     (was 0   , +38)   failed to deserialize payload for endpoint /v2/orders
12     (was 9   , +3)    retry limit reached calling upstream billing service

Verify the result

Manually seed a small app.log with a known repeated error string across two consecutive "days" (by copying yesterday's counts file) and confirm the report correctly reports the delta.

Troubleshooting

If everything shows as "new" every day, confirm YESTERDAY_COUNTS actually exists and the date arithmetic (date -u -d 'yesterday') matches the timezone your log rotation uses — a mismatch means you're always comparing against a missing file.

Next steps

Email or pipe the top-10 report through webhook-alert-dispatcher each morning, and feed raw counts through structured-json-log-emitter if you want them queryable in a dashboard instead of flat files.