Idempotent Shell Patterns — Clear exit, safe side effects, observable logs

Opening — surprising distinction

Idempotence is not the same as retryability.

Idempotence is a property of your side-effect boundary: repeated runs yield the same net effect. Retryability is a policy: when transient errors occur, you attempt the same operation again. You must decide which actions must be guarded and which may be safely retried.

# guard vs retry (conceptual)
GUARD="/var/tmp/marker.SAMPLE-CODE"
if [ -f "$GUARD" ]; then
  printf "already-applied=true\n"
  exit 0
fi
# attempt a non-destructive call (placeholder)
printf "attempt=1 cmd=call_remote --id=CASE-A\n"
# on success:
#   touch "$GUARD"            # mark completion
# human: "guard prevents re-run"
# machine: result=status=ok attempt=1
Human: check a persistent guard before mutating external systems. Machine: log key=already-applied, key=attempt, key=status.
  • Confirm boundary: what must never duplicate?
  • Mark outcome atomically where possible
  • Log attempts separately from success markers
When to use guards for external writes. When not to use
Section 1

Define what must never duplicate versus what can be reissued.

Explicitly list external side effects (POSTs, DB writes) and treat logs and temp files as safe to re-create. Use idempotent wrappers where possible and separate deterministic outputs (reports) from irreversible actions.

# marker approach (safe example)
MARKER="/tmp/task.BOX-A.done"
ACTION="echo 'notify: CASE-A'   # placeholder, non-destructive"
if [ -f "$MARKER" ]; then
  printf "status=skipped marker=%s\n" "$MARKER"
else
  printf "would-run=%s\n" "$ACTION"
  # on real run: do ACTION && touch "$MARKER"
fi
# human: explicit boundary comment
# machine: marker=present|absent
Human: separate reportable outputs from the guarded write. Machine: emit status=skipped|would-run and marker path.
  • List irreversible targets first
  • Favor markers over hidden state wherever possible
  • Keep log-only outputs repeatable
When to use during automation of external services. When not for transient local cache files.
Section 2

Prefer write-then-move semantics over in-place edits.

Writing to a temp file and renaming is a common, portable way to make the final state switch appear atomic to observers. Prefer filesystem rename as the commit signal.

# write-then-move (non-destructive demo)
OUT_TMP="$(mktemp -u "/tmp/out.XXXXXX")"
printf "result=preview\n" > "$OUT_TMP"
# validate content here (dry-check)
# commit:
# mv "$OUT_TMP" "/tmp/out-FINAL"    # commented: show intent only
# human: "use mv as atomic switch"
# machine: meta=commit=intent file=/tmp/out-FINAL
Human: produce and validate a temp artifact, then rename. Machine: log meta=commit=intent and temp path.
  • Ensure the rename target is on same filesystem
  • Fail before move if validation fails
  • Record intent so observers can detect partial runs
When to use for file commits and manifests. When not when rename semantics are not meaningful (remote APIs).
Section 3

Place and validate idempotency markers; read them first and exit cleanly.

A marker can be a file, a DB row, or an API idempotency key. Validate that a marked run completed successfully (not only that the marker exists) when possible.

# idempotency marker read pattern
MARK="/tmp/idemp.CASE-A.done"
check_marker(){
  [ -f "$MARK" ] && grep -q "success" "$MARK"
}
if check_marker; then
  printf "idempotent=true marker=%s\n" "$MARK"
  exit 0
fi
# run and on success:
# printf "success\n" > "$MARK"
# human: marker checked for success token
# machine: key=idempotent value=true
Human: confirm marker semantics include success state. Machine: emit key=idempotent and marker path.
  • Verify marker content, not just presence
  • Use monotonic naming to avoid collision
  • Expire or rotate markers if needed by policy
When to use for long-running or external writes. When not for ephemeral operations.
Section 4

Map failure categories to stable exit codes and structured messages.

Use conventional ranges: 0 success, 1–125 common failures, 126–255 for command errors or reserved conditions. More importantly, attach a stable structured message (key=value) so automation can classify failures consistently.

# normalize exit and log
die(){
  code=$1; shift
  printf "level=error code=%d msg=%s\n" "$code" "$*" >&2
  exit "$code"
}
# example usage:
# if ! do_step; then die 2 "remote-failure"; fi
# human: clear message and action hint
# machine: level=error code=2 msg=remote-failure
Human: short error text and suggested next step. Machine: consistent keys: level,code,msg,timestamp.
  • Decide which codes mean "retryable" vs "fatal"
  • Attach minimal machine keys for pipelines
  • Keep human prose short and stable
When to use for pipeline-integrated scripts. When not for quick throwaway helpers.
Section 5

Emit machine-parsable key=value lines plus an optional human summary.

Structured logs make it easy to filter and create alerts. Include a short human summary as a separate line to aid quick triage by an operator.

# structured logging example
ts="$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
printf "ts=%s level=info action=check id=CASE-A status=ok\n" "$ts"
printf "NOTICE: Completed check for CASE-A\n" >&2
# human: brief notice line
# machine: ts=... level=info action=... status=...
Human: quick summary on stderr for operators. Machine: stable keys allow parsing by grep/jq-like tools.
  • Keep keys stable (ts,level,action,id,status)
  • Emit human-friendly stderr summary where helpful
  • Avoid unstructured dumps in machine logs
When to use in production scripts. When not in throwaway experiments.
Section 6

Treat retry policy as metadata — log whether failures are retryable or fatal.

Don't bake retry loops into every script. Instead, mark failures as retryable or terminal so orchestrators can apply backoff policies. This keeps scripts simple and composable.

# signal retryability (concept)
if ! remote_call; then
  printf "level=error code=3 reason=network retryable=true\n" >&2
  exit 3
fi
# human: "network failure; orchestrator may retry"
# machine: retryable=true key present
Human: say whether to retry in the message. Machine: include retryable=true|false in log output.
  • Choose which component handles backoff
  • Log attempt counts separately
  • Keep retry metadata stable for consumers
When to use in pipeline-aware scripts. When not for purely interactive tools.
Section 7

Provide explicit --dry-run and --verify modes that never mutate state.

Design a no-op path that prints the actions it would take and a verify mode that checks current state deterministically. Tests and CI can assert stable output from these modes.

# dry-run pattern
DRY_RUN=0; VERIFY=0
# parse args (sketch)
# --dry-run sets DRY_RUN=1
if [ "$DRY_RUN" -eq 1 ]; then
  printf "DRY: would create marker %s\n" "/tmp/idemp.CASE-A.done"
  exit 0
fi
# real run would proceed here
# human: dry-run prints intent only
# machine: stable lines for CI comparison
Human: read intended effects without change. Machine: deterministic output for CI to compare.
  • Make dry-run deterministic and stable
  • Use verify to assert side-effect absence/presence
  • Log a clear prefix (DRY:) to avoid confusion
When to use for CI and safe validation. When not for interactive quick hacks.
Section 8

Capture minimal runtime context: timestamp, invocation id, guard state.

Include a small diagnostic summary at the end or on failure that an operator can read quickly. Keep it short and include essential keys for machine parsing.

# diagnostics capture
TS="$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
INV="inv.$(printf '%s' "$TS" | md5sum | cut -c1-8)"
printf "ts=%s inv=%s guard=%s\n" "$TS" "$INV" "/tmp/idemp.CASE-A.done"
# human: concise summary line
# machine: ts/inv/guard keys for triage
Human: get a one-line triage summary quickly. Machine: deterministic inv id and keys for correlation.
  • Emit a short diagnostic line on completion or error
  • Correlate logs via an invocation id
  • Keep diagnostic format stable across versions
When to use in production scripts. When not in single-run throwaways.
Closing

Quick decision map and three compact do/don't reminders.

Use these checkpoints when choosing patterns: identify the irreversible action, pick a guard or transactional commit, and expose machine-friendly logs so pipelines can act deterministically.

# quick checklist (printer-friendly)
# 1) Identify boundary => irreversible? Guard it.
# 2) Commit safely => write-then-mv or idempotent API.
# 3) Observe => key=value logs + human summary.
# human: short checklist for humans
# machine: no-op lines for scripts/CI
Do: define boundary, guard irreversible ops, use dry-run. Don't: rely on implicit side effects or unstructured logs.
  • Do label retryable failures
  • Do provide verify/dry-run modes
  • Don't conflate logs with state
Final keep patterns small and consistent; iterate policies as needs change.
Show machine diagnosis