Upgrading to 3.0

GoArchive 3.0 does not read or migrate the tracking tables of earlier versions, and it reads its configuration file strictly. Plan the upgrade as a short maintenance step: finish the running jobs with your current version, then switch.

The short version #

1. Finish every job with your current version, then drop its tracking tables (Moving from 2.x ). GoArchive 3.0 never reads or migrates them.
2. Rewrite the configuration: every job needs job_type, verification is one value, and unknown keys are refused (Configuration changes ).
3. Grant TRIGGER directly on the job tables: on the source for archive and purge, and on the destination for archive and copy-only. The 2.x grant recipes lack it, and preflight refuses the job without it (Permissions ).
4. Drop --force and --skip-verify from your scripts, rename --skip-validate-preflight, and check how the scripts read exit codes (Command-line changes ).
5. Run goarchive validate. Preflight in 3.0 checks keys and relations more strictly than 2.x did.

A 2.x configuration file fails to load with an error that points to the upgrade guide, and 2.x tracking tables are refused at startup.

Moving from 2.x #

The tracking tables in 3.0 are new: a typed log per job and a job binding, with no checkpoint (Tracking Tables ). A tracking schema written by an earlier version is refused, and nothing is migrated.

1. Finish every job with your current version, so that no log row is left pending or copied. Retire a job you cannot finish, as your current version’s documentation describes.

2. Stop every GoArchive process that uses the tracking schema.

3. Drop the old tracking tables in job_schema, as a DBA. GoArchive’s own account holds no DROP privilege.

   -- Generate the DROP statements for every log table (substitute your job_schema for
   -- goarchive), review them, run them.
   SELECT CONCAT('DROP TABLE `', table_schema, '`.`', table_name, '`;')
   FROM information_schema.tables
   WHERE table_schema = 'goarchive' AND table_name LIKE 'archiver_job_log%';
   
   DROP TABLE goarchive.archiver_job;
   DROP TABLE goarchive.goarchive_meta;
   

   Export the old logs first if the record of what was archived has compliance value. Alternatively, point destination.job_schema at a new, empty schema.

4. Run 3.0. The first run labels the schema and registers each job afresh. Every run starts at the beginning of the job’s where range. For archive and purge, the rows already processed are gone from the source. A copy-only job would copy everything again, so point it at an empty destination, or give it a where that starts after what the earlier version copied.

What else is new against 2.x #

• Destination tables must be InnoDB. archive and copy-only refuse a destination table on another engine, and validate and dry-run report it. A destination whose columns are in a different order from the source is accepted, with one warning per table.
• Archive recovery checks the destination before it deletes. When a run resumes interrupted work, each row of a logged root must already be in the destination before any of it is deleted; otherwise the run stops and deletes nothing. Recovery reads the destination even under verification: none, so the archive account needs SELECT on the destination tables. An archive job is also bound to its destination database while its log holds copied roots.
• Large rows are copied. GoArchive reads each server’s max_allowed_packet when it connects. Only a single value larger than the destination’s packet, or rarely a whole INSERT, fails its batch, with an error that names the table and the remedy.
• TRIGGER must be provable on every job table, granted directly to the account. A TRIGGER privilege held only through a role is refused. --force-triggers does not cover these checks.
• A self-referencing ON DELETE CASCADE or SET NULL on a job table is refused for archive and purge, because deleting a batch row would change rows of that table outside the batch.
• A job column in an unsupported character set is refused. Every session runs in utf8mb4, and preflight refuses a column whose character set does not survive that session exactly (Character Sets ).
• A stopped job with unfinished work holds its tables. Another job on the same source data whose tree shares one of those tables is refused until the stopped job’s next run finishes its work (Concurrency and locking ).
• Output and exit codes: a run prints its report on stdout only with --progress, warnings go to stderr, and the exit code tells a stop (3) and a run held by another live run (4) apart from a failure (1) (Output, Progress and Exit Codes ).
• Warnings before a run: verification: count, verification: none, and JSON columns under either mode each print a warning, because those modes do not compare copied values.
• The container image runs as UID 65532 from /work, not as root. A configuration file mounted under /root/ is no longer readable; mount it under /work/ (Installation ).
• Log files are created readable by their owner only (0600) on Unix, and a run’s log ends with one entry that says how the run ended, with its counts.

Configuration changes #

The file is read strictly (Configuration ).

• Every job needs job_type: archive, purge or copy-only. A job runs only under that command.

• primary_key, foreign_key and references may be lists, for composite keys. primary_key must name exactly the table’s PRIMARY KEY columns, in key order. references names the parent’s key columns, paired with foreign_key by position, and may be omitted only when the parent key has one column.

• verification is one value: sha256 (the default), count or none, globally or per job. The 2.x block (method, skip_verification) is refused:

  2.x	3.0
  method: count	verification: count
  method: sha256	verification: sha256 (the default)
  skip_verification: true	verification: none

• Each job type accepts only the settings it uses. A purge job with verification, or a copy-only job with batch_delete_size or delete_sleep_seconds, is refused. resume is a copy-only setting.

• Unknown keys are refused. Key names are exact and lowercase: Root_Table is refused. A helper key such as x-defaults: is refused too, so a YAML anchor shared by merge keys must live under a known key, for example a real job.

• Job names keep their case. The name on the command line must match the file exactly, and two names that differ only in case are refused. Names must be printable ASCII without spaces, must not start with -, and can be at most 255 characters.

• A table may appear once in a job.

• One YAML document per file: a second document after --- is refused rather than ignored.

• The 2.0 replica keys are refused:

  2.0	3.0
  replica: (one server)	replication.servers: (a list)
  replica.host / port / user / password	replication.servers[].host / port / user / password
  replica.replication_channel: "billing"	replication.servers[].channels: ["billing"]
  replica.replication_channel: ""	replication.servers[].channels: [""], the default channel only; omitted or [] checks every channel
  safety.lag_threshold	replication.seconds_behind_source_within
  safety.check_interval	replication.check_interval

• A new log file is readable by its owner only on Unix (0600; 2.x created it 0644). On Windows it inherits its folder’s permissions. An existing file keeps its mode.

Command-line changes #

Limits removed or changed #

Coming from a release before 2.2 #

GoArchive 3.0 keeps what 2.0 to 2.2 introduced. If you skip those releases:

• Grants must be provable: each privilege granted directly to the account, and PROCESS on the source for archive and purge (Permissions ).

• Sessions are UTC: where runs in a UTC session, so a DATETIME column compared with NOW(), a TIMESTAMP column compared with a bare literal, and CURDATE() boundaries fall at UTC rather than in the server’s time zone. last_heartbeat_at is UTC: measure its age with UTC_TIMESTAMP().

• TIMESTAMP values archived before 2.2 may be wrong when the source and destination sessions ran in different time zones: each copy took the source’s wall-clock text, and the destination read it in its own zone. A DBA can repair them once:

  -- 1. Back the table up.
  -- 2. A non-zero count means a zone name is not recognised and the UPDATE would NULL
  --    the column: stop.
  SET time_zone = '+00:00';
  SELECT COUNT(*) FROM archive.orders
   WHERE <only rows archived before 2.2>
     AND ts IS NOT NULL
     AND CONVERT_TZ(CONVERT_TZ(ts, '+00:00', '<zone the old DESTINATION sessions used>'),
                    '<zone the old SOURCE sessions used>', '+00:00') IS NULL;
  
  -- 3. Repair, exactly once per column.
  UPDATE archive.orders
     SET ts = CONVERT_TZ(CONVERT_TZ(ts, '+00:00', '<zone the old DESTINATION sessions used>'),
                         '<zone the old SOURCE sessions used>', '+00:00')
   WHERE <only rows archived before 2.2>;
  

  Name the zones as the old sessions saw them (SYSTEM means the server’s operating-system zone). A second run corrupts the values again. Values that fell in a daylight-saving fall-back hour cannot be recovered.