Build a CI-Style Staged Pipeline Runner

What you will learn

Build a small CI-style pipeline runner in plain Bash: named stages executed in order, fail-fast on the first broken stage, per-stage log capture, and a packaged artifact at the end — no CI platform required.

Before you begin

This is meant for a self-contained build/deploy script that needs pipeline structure without adopting a full CI system, or for running the same stages locally that a real CI later wraps.

1. Define stages as an ordered list of function names

#!/usr/bin/env bash
set -Eeuo pipefail

STAGE_LOG_DIR="$(mktemp -d)"
trap 'echo "logs kept at: ${STAGE_LOG_DIR}"' EXIT

stage_lint()  { shellcheck ./scripts/*.sh; }
stage_test()  { ./run-tests.sh; }
stage_build() { tar -czf "${STAGE_LOG_DIR}/artifact.tar.gz" ./dist; }
stage_smoke() { ./dist/bin/myapp --version; }

STAGES=(lint test build smoke)

Naming each stage as stage_<name> and listing the names separately keeps the pipeline definition (the order) decoupled from the implementation (the function body) — reordering or removing a stage is a one-line change.

2. Run each stage, capturing output and stopping on first failure

run_pipeline() {
    local stage fn log start_ts end_ts

    for stage in "${STAGES[@]}"; do
        fn="stage_${stage}"
        log="${STAGE_LOG_DIR}/${stage}.log"
        start_ts="$(date +%s)"

        echo "=== [${stage}] starting ==="
        if "$fn" > "$log" 2>&1; then
            end_ts="$(date +%s)"
            echo "=== [${stage}] ok ($((end_ts - start_ts))s) ==="
        else
            end_ts="$(date +%s)"
            echo "=== [${stage}] FAILED ($((end_ts - start_ts))s) ===" >&2
            echo "--- last 20 lines of ${stage}.log ---" >&2
            tail -20 "$log" >&2
            return 1
        fi
    done
}

Redirecting each stage's own stdout/stderr into a per-stage log — while still printing a live starting/ok/FAILED line to the console — gives a readable top-level narrative during the run and full detail on disk for whichever stage actually breaks.

3. Summarize and exit with the right code

if run_pipeline; then
    echo "pipeline succeeded, artifact: ${STAGE_LOG_DIR}/artifact.tar.gz"
    exit 0
else
    echo "pipeline failed" >&2
    exit 1
fi

Example output

=== [lint] starting ===
=== [lint] ok (2s) ===
=== [test] starting ===
=== [test] FAILED (14s) ===
--- last 20 lines of test.log ---
FAIL: test_order_totals (expected 42.00, got 41.99)
pipeline failed
logs kept at: /tmp/tmp.9fKq2L

Verify the result

Make stage_test fail deliberately and confirm stage_build and stage_smoke never run — the pipeline must stop at the first failure, not continue and mask it.

Troubleshooting

If a stage's failure doesn't stop the pipeline, check that stage_build's tar call isn't being run in a context where set -e is suppressed (e.g. inside if stage_build; then conditions already handle their own exit status correctly here — the bug is usually a stage function backgrounding part of its own work with & and losing the exit code).

Next steps

Parameterize STAGES via getopts-cli-argument-parser so a caller can run a subset (--only lint,test), and route the failure block through webhook-alert-dispatcher.