Archiving Data

The archive workflow copies data to the destination, confirms it arrived, and then removes it from the source. It is the complete cycle.

Running it #

goarchive archive -c archiver.yaml --job archive_old_orders

What happens #

GoArchive first checks the environment: that the tables exist on both sides, that their structures are compatible, that your account holds the necessary privileges, and that the declared relationships match the database. Only then does it begin.

Then, for each batch:

1. Select root rows matching the job’s filter.
2. Discover every row belonging to them.
3. Copy them to the destination, parents first.
4. Verify the copy.
5. Remove them from the source, children first.
6. Record progress.

Between batches it waits for sleep_seconds, checks replica lag if a replica is configured, and checks whether a pause has been requested.

Before your first run #

Work through the three inspection commands in order:

goarchive validate -c archiver.yaml
goarchive dry-run  -c archiver.yaml --job archive_old_orders
goarchive archive  -c archiver.yaml --job archive_old_orders

validate confirms the environment. dry-run shows the filter and the number of rows each table would contribute. Neither changes anything.

Recommended settings #

verification:
  method: sha256

processing:
  batch_size: 1000
  batch_delete_size: 500
  sleep_seconds: 1

sha256 verification is the stronger guarantee and makes an interrupted job straightforward to resume.

If the run is interrupted #

Run the same command again. GoArchive reads its own progress records and continues from where it stopped, repeating only the work that did not finish. See Checkpoints and Resume .

Tables with triggers #

If a source table has a trigger that fires on deletion, GoArchive reports it and waits for you to confirm, because the trigger will run as part of the removal and its effects are outside what the job describes.

goarchive archive -c archiver.yaml --job archive_old_orders --force-triggers

Review what those triggers do before passing the flag.

Confirming the result #

-- rows now in the archive
SELECT COUNT(*) FROM archive.orders;

-- rows remaining in the source that matched the filter
SELECT COUNT(*) FROM production.orders
 WHERE created_at < DATE_SUB(NOW(), INTERVAL 2 YEAR);