Output, Progress and Exit Codes

GoArchive writes a command’s result on stdout, diagnostics on stderr, and every log entry to the log, as logging configures it. Nothing reaches stdout that you did not ask for, and an error always reaches stderr.

Streams #

Three logging setups #

• A log file (output: <path>), the setup for scheduled runs. Without --progress, a run is silent: it prints nothing when it completes or stops, and on a failure prints a summary and the error on stderr:

  Job archive_old_orders (archive): failed in 61.240s. Log: /var/log/goarchive/orders.log
  Error: engine: …
  

  A failure before startup completes (connecting, preflight, locks, registration) says failed before the first batch. instead of a duration. With --progress, the start line, the progress lines and the closing block go to stdout, and the log’s warnings and errors go to stderr as text lines, with live events whatever the level: a replication hold, the replica recovering, a stop ending a hold, a pause and its end, recovery. file_only: true keeps every log entry off the console; warnings and errors still reach stderr.

• The log on the console as text (output: stdout or stderr, format: text): the log stream is plain text, with no colour codes. --progress adds its lines and the closing block on stdout.

• The log on the console as JSON (output: stdout or stderr, format: json), the default and the setup for containers: every line of the log stream is one JSON object. --progress writes Run starting, progress and the closing entry there instead of printing lines, and the closing entry carries the counts. On a failure with output: stdout, one plain Error: line goes to stderr; with output: stderr, the error is the closing entry itself. An error found before the log exists (a flag, the configuration file, an unknown job) is always a plain Error: line.

validate and dry-run print their reports as text on stdout in every setup. In a container, set logging.output: stderr for them, so stdout carries only the report and stderr only the log.

Progress display #

A bare --progress reports every 30 seconds. --progress=10s or --progress=1m30s sets the interval, and a bare integer is seconds (minimum one second). The value must be attached with =: --progress 10s is refused with that hint.

Job archive_old_orders (archive): starting. Log: /var/log/goarchive/orders.log
progress: roots  4000/~12000 ( 33.3%)  remaining=8000   copied_rows=21640      deleted_rows=21640      elapsed=42s      eta=1m18s
progress: roots  8000/~12000 ( 66.7%)  remaining=4000   copied_rows=43310      deleted_rows=43310      elapsed=1m12s    eta=36s
progress: roots 12000/~12000 (100.0%)  remaining=0      copied_rows=64920      deleted_rows=64920      elapsed=1m49s    eta=0s

• Columns have fixed widths, so successive lines line up. With the log as JSON on the console, each line is instead a progress log entry with the fields roots_done, roots_total, percent, remaining, copied_rows, deleted_rows, elapsed_seconds and eta, and, while paused, paused.
• The total is an estimate. roots, the percentage, remaining and the ETA count root rows. The total, marked with ~, is the optimizer’s estimate of the where range, read once after recovery; nothing is counted, so --progress adds no pass over the source. It is close when an index serves the where, and can be far off in either direction when the where filters each row with a subquery or a function.
• Limits: the percentage caps at 100 and remaining at zero. The ETA shows calculating until a root is done and unknown past the estimate, and a run that finishes shows its done count as the total. dry-run counts the range exactly.
• If the estimate cannot be read, the run fails before its first batch (engine: --progress estimate on <table>: …). Run without --progress to proceed.
• The final line: once the first batch has started, one final line is printed on every exit, and a run that ends naturally shows remaining=0. A run that stops or fails during startup or recovery prints no progress line.

The closing block #

With --progress, a run ends with a closing block on stdout:

Job archive_old_orders (archive): completed in 109.412s
  roots: 12000 in 12 batches

  table           copied  deleted  verified (sha256)
  orders           12000    12000              12000
  order_items      40920    40920              40920
  order_payments   12480    12480              12480

The status is completed, stopped before the end of the where range: run the job again to continue, or failed, with the error on stderr. Purge shows deleted, copy-only copied and verified; under verification: none there is no verified column. Recovered roots, skipped roots and accepted conversions are listed under roots.

Exit codes #

A stop that meets an error exits 1. In a scheduled script:

goarchive archive -c archiver.yaml --job archive_old_orders
case $? in
  0|3) ;;                     # done, or stopped: the next run continues
  4)   exit 0 ;;              # another run holds the work: try later
  *)   exit 1 ;;              # failed or misused: read stderr and the log
esac

Containers #

Keep the default logging, JSON to stdout, and the container’s log stream carries the run with no file or mount. A foreground docker run returns GoArchive’s exit code. For a detached container, the code is the number docker wait <name> prints. docker stop sends SIGTERM, a graceful stop: the batch in flight finishes and the run exits 3, if it gets there before the stop deadline (10 seconds by default; then the process is killed, and the next run recovers what the log holds). Set docker stop --time, or stop_grace_period in Compose, above one batch’s duration. See Installation for the image’s user and mounts.