Documentation menu

Version 3.0.0

Replication Gating

Hold deletes while any monitored replica is lagging, stopped or unreachable, and continue the same run when they recover.

Deletes on the source travel to its replicas. With replica monitoring enabled, GoArchive checks every configured replica before every delete chunk, recovery’s included, and before a purge batch logs its roots, and holds the job while any of them is unhealthy. The same run continues once every replica is healthy again: a hold is a pause, not a failure.

replication:
  enabled: true
  seconds_behind_source_within: 10
  check_interval: 5
  servers:
    - host: replica-1.internal
      user: monitor
      password: change_me

Every option is described under Database Connections . Each monitored account needs REPLICATION CLIENT on its server.

Copy-only never deletes, so it is never held. To throttle a copy-only job, use sleep_seconds, or pause it with the sentinel file .

When a replica is unhealthy

A replica is unhealthy when:

  • it is unreachable;
  • replication is not configured or not running;
  • it is further behind than seconds_behind_source_within seconds;
  • its status cannot be read, a privilege error included. An account lacking REPLICATION CLIENT holds the job rather than being skipped.

By default, every replication channel counts; channels narrows that per server.

What you see while it holds

One WARN line per unhealthy server per check, with the reason, and from the second check on, the time held so far:

replication hold: server replica1.internal:3306 unhealthy (channel <default>: lag=42s tolerance=10s); job held, retrying in 5s
  • Healthy servers stay silent.
  • A recovered server logs one INFO line, and the last one to recover logs another with the total hold.
  • With --progress and a log file, the hold and the recovery lines also reach stderr.
  • If the reason changes while a server is still unhealthy, the held time keeps accumulating.

A hold keeps no source session open, so it can last as long as the replicas need. A first SIGTERM or SIGINT ends it without waiting for them, with one more WARN line:

replication hold ended by a stop request; job held 2h14m5s

Check caching and detection delay

A passing result is cached for cache_ttl seconds (15 by default), so a fast loop does not query healthy replicas before every chunk. Failures are never cached. For up to cache_ttl seconds after a pass, a replica that has just become unhealthy is not noticed. Set cache_ttl: 0 to check every time.

Pacing deletes for replicas

Holding is the safety net; pacing keeps the replicas from falling behind in the first place. Lower batch_delete_size and set delete_sleep_seconds when delete volume is what makes replicas lag (Batch Processing ).