Batch Processing

GoArchive never processes a whole table in one operation. Work is divided into batches, and each batch completes fully before the next begins.

The batch cycle #

Every batch follows the same five steps:

1. Select a set of root rows matching the job’s filter.
2. Discover every row belonging to them, following the declared relations.
3. Copy the result to the destination, parents first.
4. Verify that the copy matches the source.
5. Remove the source rows, children first.

Progress is recorded at the end of each batch. A run that stops partway has completed whole batches, never half of one.

Sizing a batch #

batch_size sets how many root rows a batch covers. It is also the unit used to read and write every table in that batch, root and children alike.

processing:
  batch_size: 1000

Larger batches mean fewer round trips and more throughput. Smaller batches mean less memory and shorter operations against the server. A thousand is a reasonable starting point; adjust after observing a real run.

dry-run checks your chosen size against the destination’s limits and tells you if it is too large for a particular table.

Sizing deletions separately #

Removal is throttled independently of copying:

processing:
  batch_delete_size: 500

This controls how many rows each delete statement covers. Lowering it produces smaller, more frequent operations, which reduces the volume of replication traffic generated at once.

Pacing #

Two independent settings introduce pauses, addressing two different pressures.

processing:
  sleep_seconds: 1           # between batches
  delete_sleep_seconds: 0    # between delete operations

sleep_seconds pauses between batches. Use it to keep general load on the servers within a range you are comfortable with.

delete_sleep_seconds pauses between delete operations within a batch. Use it when the constraint is how fast changes reach your replicas rather than load on the source itself.

Both accept fractional values, and both can be set globally or per job.

Replica awareness #

When a replica is configured, GoArchive checks its lag before each batch and waits while it exceeds your threshold.

replica:
  enabled: true
  host: replica-db.internal
  user: repl_user
  password: change_me

safety:
  lag_threshold: 10
  check_interval: 5

Processing continues automatically once the replica catches up. Nothing is lost and nothing needs restarting.

Pausing a running job #

A job can be paused without stopping it. Name a file in the configuration:

processing:
  sentinel_file: /var/run/goarchive/pause.flag

While that file exists, GoArchive waits before starting each batch, checking once per second. Remove the file and processing resumes.

touch /var/run/goarchive/pause.flag   # pause
rm /var/run/goarchive/pause.flag      # resume

This is useful when something else needs the database’s attention. The process stays alive, keeps its place, and picks up where it left off.