Processing and Safety

Batch sizes, pacing, replica thresholds, verification, and logging.

These blocks control how a job runs: how much it does at a time, how quickly, how it confirms its work, and what it reports.

Processing

processing:
  batch_size: 1000
  batch_delete_size: 500
  sleep_seconds: 1
  delete_sleep_seconds: 0
  sentinel_file: ""

Setting	Description	Default
`batch_size`	Root rows per batch, and the read/write unit for every table	`1000`
`batch_delete_size`	Rows per delete operation	`500`
`sleep_seconds`	Pause between batches	`1`
`delete_sleep_seconds`	Pause between delete operations	`0`
`sentinel_file`	Path whose existence pauses the run	empty

sleep_seconds moderates overall load on the servers. delete_sleep_seconds moderates how quickly changes are generated for replicas. Both accept fractional values.

sentinel_file is the pause switch described in Batch Processing . While the file exists, GoArchive waits before each batch; remove it and the run continues.

Safety

safety:
  lag_threshold: 10
  check_interval: 5
  disable_foreign_key_checks: false

Setting	Description	Default
`lag_threshold`	Replica lag, in seconds, above which processing pauses	`10`
`check_interval`	How often replica lag is checked, in seconds	`5`
`disable_foreign_key_checks`	Relax destination constraint checking during the copy	`false`

The first two apply when a replica is configured in Database Connections .

disable_foreign_key_checks is useful when the destination holds only the tables in your job, while their constraints refer to reference tables that were not copied. It applies to the copy operation alone and is restored immediately afterwards.

Verification

verification:
  method: sha256
  skip_verification: false

Setting	Description	Default
`method`	`count` or `sha256`	`count`
`skip_verification`	Skip the verification step	`false`

See Verification for how to choose.

Logging

logging:
  level: info
  format: json
  output: stdout
  file_only: false

Setting	Description	Default
`level`	`debug`, `info`, `warn`, or `error`	`info`
`format`	`json` or `text`	`json`
`output`	`stdout`, `stderr`, or a file path	`stdout`
`file_only`	Write only to the file, not also to the console	`false`

Naming a file path writes to that file and mirrors output to the console. Set file_only: true for the file alone. Every entry carries the job name, so several jobs can share a destination and remain distinguishable.

GoArchive does not rotate its own log files; they are opened in append mode, so an external rotation tool can manage them.

Per-job overrides

processing, verification, and logging may each be overridden by an individual job:

processing:
  batch_size: 1000
  sleep_seconds: 1

verification:
  method: count

jobs:
  archive_old_orders:
    root_table: orders
    primary_key: id
    where: "created_at < DATE_SUB(NOW(), INTERVAL 2 YEAR)"
    processing:
      batch_size: 500
      sleep_seconds: 0
    verification:
      method: sha256
    logging:
      output: /var/log/goarchive/orders.log
      file_only: true

Settings a job does not mention are inherited from the global block.

Processing #

Safety #

Verification #

Logging #

Per-job overrides #

Processing

Safety

Verification

Logging

Per-job overrides