Documentation menu

Version 3.0.0

Character Sets

The character sets GoArchive supports and refuses, and the session character set every connection uses.

GoArchive reads and writes every text value through a utf8mb4 session, and some character sets do not convert to and from utf8mb4 exactly: two characters can read as one, or a character can read as ?. GoArchive therefore refuses a job whose columns use such a set, before any row is read.

The session character set

Every connection is set to utf8mb4, collated utf8mb4_general_ci, after authentication, so a server’s init_connect setting cannot narrow it. Each source and destination connection then proves its session settings when it opens (SESSION_CHARSET_CHECK ). Comparisons in a job’s where and in GoArchive’s key SQL use that collation.

Columns are checked

Before any row is read, preflight refuses a job column the run reads or writes as text whose character set is not supported (COLUMN_CHARSET_CHECK ):

Job typeColumns checked
archive, copy-onlyevery column of every job table on the source, and every destination column of the same name
purgethe primary-key and foreign_key columns of every job table, on the source

ENUM and SET columns are checked. Numbers, temporal types, binary strings and JSON have no character set and are not. There is no bypass: both checks are enforced when preflight is bypassed, because the loss would be silent and could not be found after the run.

Supported character sets

Character setNote
utf8mb4The session’s own character set
utf8mb3
asciiBytes 0x80–0xFF cannot be stored, so none reaches a run
latin1
latin2
latin5
latin7
cp850
cp852
cp866
keybcs2
koi8r
koi8u
macce
macroman
utf16Every Unicode character, surrogate pairs included
utf16leEvery Unicode character, surrogate pairs included
utf32Every Unicode character
binaryListed for completeness: a binary string has no character set

Refused character sets

Every other character set is refused:

Character setWhy it is refused
sjisTwo characters read as one; some codes read as ?
cp932Two characters read as one (the NEC and IBM duplicates); some codes read as ?
eucjpmsTwo characters read as one (the NEC and IBM duplicates); some codes read as ?
ujisTwo characters read as one (a full-width backslash and tilde); some codes read as ?
armscii8Six punctuation bytes read as their ASCII look-alikes
tis620Some codes read as U+FFFD
big5Some codes read as ? or U+FFFD
euckrSome codes read as ?
gb2312Some codes read as ?
gbkSome codes read as ?
gb18030Some codes read as ?
cp1250Unmapped bytes read as ?
cp1251Unmapped bytes read as ?
cp1256Unmapped bytes read as ?
cp1257Unmapped bytes read as ?
dec8Unmapped bytes read as ?
geostd8Unmapped bytes read as ?
greekUnmapped bytes read as ?
hebrewUnmapped bytes read as ?
hp8Unmapped bytes read as ?
swe7Unmapped bytes read as ?
ucs2Lone surrogates read as bytes that are not UTF-8 and cannot be bound back

A refused column is converted to utf8mb4 on both servers, or its table is left out of the job.