Build a Self-Managing Log Rotation Pipeline

What you will learn

Build a log rotation pipeline that compresses aged log files, verifies each archive with a checksum, and prunes anything past a retention window — without truncating a file a process still has open for writing.

Before you begin

Target Bash 4+ on Linux with gzip, sha256sum, and find available. Run against a scratch log directory first; a rotation bug that deletes live logs is expensive to discover in production.

1. Define the guardrails up front

#!/usr/bin/env bash
set -Eeuo pipefail
IFS=$'\n\t'

LOG_DIR="/var/log/myapp"
ARCHIVE_DIR="${LOG_DIR}/archive"
RETENTION_DAYS=14
MAX_AGE_MINUTES=1440   # only rotate files older than 24h

set -Eeuo pipefail turns unset variables and unchecked failures into hard stops instead of silent corruption — critical in anything that deletes files.

2. Rotate only files that are safe to touch

rotate_file() {
    local src="$1"
    local base ts dest

    base="$(basename "$src")"
    ts="$(date -u +%Y%m%dT%H%M%SZ)"
    dest="${ARCHIVE_DIR}/${base}.${ts}.gz"

    # Skip anything a process still has open for writing.
    if lsof -- "$src" >/dev/null 2>&1; then
        echo "skip (in use): $src" >&2
        return 0
    fi

    gzip -c -- "$src" > "$dest"
    sha256sum -- "$dest" > "${dest}.sha256"
    : > "$src"   # truncate in place; the process keeps its fd, next write starts clean
    echo "rotated: $src -> $dest"
}

Truncating with : > "$src" (rather than deleting the file) keeps the original inode alive, so a process holding it open keeps writing to the same path instead of an orphaned file descriptor nobody reads.

3. Walk the directory and prune old archives

mkdir -p "$ARCHIVE_DIR"

find "$LOG_DIR" -maxdepth 1 -type f -name '*.log' -mmin +"$MAX_AGE_MINUTES" -print0 |
    while IFS= read -r -d '' f; do
        rotate_file "$f"
    done

find "$ARCHIVE_DIR" -type f -name '*.gz' -mtime +"$RETENTION_DAYS" -print0 |
    while IFS= read -r -d '' old; do
        rm -f -- "$old" "${old}.sha256"
        echo "pruned: $old"
    done

-print0 / read -r -d '' handles filenames with spaces or newlines correctly — a plain for f in $(find ...) breaks on those.

Example output

rotated: /var/log/myapp/app.log -> /var/log/myapp/archive/app.log.20260919T030000Z.gz
skip (in use): /var/log/myapp/access.log
pruned: /var/log/myapp/archive/app.log.20260901T030000Z.gz

Verify the result

Confirm each archive's checksum: sha256sum -c app.log.*.gz.sha256. Tail the live log immediately after a rotation to confirm the application is still writing to the same path with no gap.

Troubleshooting

If archives never age out, check that -mtime is reading the archive's own mtime and not an inherited one from cp -p. If lsof isn't installed, fall back to checking /proc/*/fd on Linux, or accept a short quiesce window instead.

Next steps

Wire this into flock-cron-concurrency-guard so a slow rotation run can never overlap itself, and forward the rotated/pruned lines through structured-json-log-emitter for ingestion by your log pipeline.