What you will learn

React to new or modified files in a drop folder the instant they land, using inotifywait, instead of polling the directory on a timer and adding latency or wasted CPU.

Before you begin

Install inotify-tools (apt install inotify-tools / dnf install inotify-tools). inotify watches are a finite kernel resource per user — this matters if you're watching thousands of directories at once, not for a single drop folder.

1. Watch for fully-written files, not partial uploads

#!/usr/bin/env bash
set -Eeuo pipefail

WATCH_DIR="/data/incoming"
PROCESSED_DIR="/data/processed"
FAILED_DIR="/data/failed"

mkdir -p "$PROCESSED_DIR" "$FAILED_DIR"

inotifywait -m -e close_write --format '%f' "$WATCH_DIR" |
while IFS= read -r filename; do
    process_file "$WATCH_DIR/$filename"
done

Watching close_write (not create) is the key detail: create fires the instant a file appears, which for a large upload can be before it's finished writing. close_write only fires once whatever wrote it has closed the file handle.

2. Process defensively — the file could still vanish

process_file() {
    local path="$1"
    local name; name="$(basename "$path")"

    # Race guard: another watcher instance, or the uploader retrying, could have moved it already.
    [[ -f "$path" ]] || { echo "skip (already gone): $name"; return; }

    if validate_and_import "$path"; then
        mv -- "$path" "${PROCESSED_DIR}/${name}"
        echo "processed: $name"
    else
        mv -- "$path" "${FAILED_DIR}/${name}"
        echo "failed validation, moved to failed/: $name" >&2
    fi
}

validate_and_import() {
    local path="$1"
    [[ "$path" == *.csv ]] || return 1
    head -c1 "$path" >/dev/null 2>&1 || return 1   # basic readability check
    # ... real import logic here ...
    return 0
}

Moving the file out of the watch directory (rather than deleting or leaving it) prevents close_write from firing again on a later unrelated touch, and gives you an audit trail of what was accepted versus rejected.

3. Make the watcher itself supervisable

trap 'echo "watcher stopped"; exit 0' SIGTERM SIGINT
echo "watching ${WATCH_DIR} (pid $$)"

Run this under a process supervisor (systemd, pidfile-heartbeat-supervisor) — inotifywait -m runs forever and needs the same lifecycle management as any long-running daemon.

Example output

watching /data/incoming (pid 4021)
processed: batch-2026-09-19.csv
failed validation, moved to failed/: corrupt-upload.csv

Verify the result

Copy a large file into the watch directory with cp (which can be slow for big files) and confirm processing only starts after the copy fully completes, not partway through.

Troubleshooting

If events stop firing after a while, check cat /proc/sys/fs/inotify/max_user_watches — a directory watch on a path with heavy churn can exhaust the limit; watching a single directory non-recursively (no -r) avoids this in most drop-folder setups.

Next steps

Feed processed/failed outcomes into structured-json-log-emitter, and alert on repeated validation failures via webhook-alert-dispatcher.