Tracking Tables

GoArchive keeps all its tracking state on the destination server, in the schema named by destination.job_schema, by default the destination database itself. The tables are the record of what left production: nothing in them is cleaned up automatically.

GoArchive creates the tables but never the schema, and its account holds no DELETE or DROP there: every deletion is a DBA’s act (Tracking Table Maintenance ). The examples below use a job_schema named goarchive and job id 42; substitute yours.

SELECT id, job_name, root_table, job_type FROM goarchive.archiver_job;
-- job id 42  ->  goarchive.archiver_job_log_42

The job registry (archiver_job) #

CREATE TABLE IF NOT EXISTS archiver_job (
  id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY,
  job_name VARCHAR(255) CHARACTER SET ascii COLLATE ascii_general_ci NOT NULL,
  root_table VARCHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
  job_type VARCHAR(16) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
  job_status TINYINT NOT NULL DEFAULT 0,
  source_server_uuid VARCHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
  source_database VARCHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
  destination_database VARCHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
  destination_server_uuid VARCHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL DEFAULT '',
  config_fingerprint TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
  root_key_columns TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  last_heartbeat_at DATETIME NULL,
  UNIQUE KEY uk_job_name (job_name),
  KEY idx_status (job_status)
) ENGINE=InnoDB

The source and destination columns and config_fingerprint are the job’s binding: the facts that give its log rows their meaning.

The job log (archiver_job_log_<id>) #

Created when the job is registered, and typed by the root key: one column k1…kN per root key column, in key order, with the column’s own type, charset and collation. A TIMESTAMP is stored as DATETIME of the same precision, so no session time zone can shift it. For a root key (order_id BIGINT UNSIGNED, line_no INT):

CREATE TABLE IF NOT EXISTS archiver_job_log_42 (
  `k1` bigint unsigned NOT NULL COMMENT 'order_id',
  `k2` int NOT NULL COMMENT 'line_no',
  ga_status TINYINT NOT NULL,
  ga_updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`k1`, `k2`),
  KEY idx_ga_status (ga_status)
) ENGINE=InnoDB

Each comment names the root column, for people; GoArchive never reads it. The table must stay InnoDB, with PRIMARY KEY (k1…kN) and the two ga_ columns as created: a log whose shape changed is refused at startup. Only roots are logged, never child rows.

Status values #

Each status is written in the transaction that does the work it records. What a rerun does with them is described under Crash Recovery and Resume .

Startup checks #

Before anything is recovered or run, every run checks its job against the registry. These are bookkeeping integrity checks: no flag skips them.

1. The job is fixed. A different job_type or root_table under an existing name, or a name that differs from a stored one only in letter case, is refused: use a new name, or retire the job .

2. The root key is fixed. root_key_columns and the log’s k1…kN must match the root table’s current primary key: names, order, types, charsets and collations. A changed key is refused, since the log cannot record it: retire the job or use a new name.

3. The binding. While the job has pending work (archive and purge: any copied or deleting root; copy-only: any completed root, whatever its resume setting), a change to the parts that work depends on is refused, naming the parts and the number of roots:

   Job type	Compared
   archive	source server and database, relations, destination database
   purge	source server and database, relations, where
   copy-only	source server and database, relations, destination database

   The refusal lists three ways out: restore the original configuration or source and run once; abandon the roots ; or, after a source failover, re-bind the job . With no pending work, the stored binding is replaced by the current one, so changes between clean runs are free.

Registration takes two steps, because the log is named by the job’s id: the job row is written with an empty root_key_columns, then the log is created, and only then are the key’s names written. A run that finds an empty root_key_columns completes the registration. A table that already exists under the new log’s name and holds rows, or has another shape, is never adopted: the job stays unregistered until a DBA drops that table, after confirming no job uses it, or restores archiver_job.

Lifecycle #

1. First run in the schema: goarchive_meta and archiver_job are created under the schema lock, and a fresh schema is labelled 3.0.
2. First run of a job: the job is registered and its log created.
3. During a run: job_status = 1, the heartbeat refreshes, and log rows are written per batch.
4. The end of a run: job_status returns to 0, or 3 after an error. A crash or kill leaves 1 with a stale heartbeat. A job whose log still holds copied or deleting roots, in any status, holds its tables against overlapping jobs until its own rerun finishes them.

The schema label #

CREATE TABLE IF NOT EXISTS goarchive_meta (
  id TINYINT NOT NULL PRIMARY KEY,
  schema_version VARCHAR(16) NOT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB

One row, id = 1. The value is the tracking-schema label, not the program version: a release changes it when it changes the tables’ layout or a column’s meaning. GoArchive 3.0 writes and recognises exactly one label, 3.0. At startup it labels a fresh schema and refuses every other state; there is no automatic migration.

When a schema is refused, stop every GoArchive process that uses it, whatever its version, then follow the row that matches the refusal: