Character Sets
The character sets GoArchive supports and refuses, and the session character set every connection uses.
GoArchive reads and writes every text value through a utf8mb4 session, and some character sets do not convert to and from utf8mb4 exactly: two characters can read as one, or a character can read as ?. GoArchive therefore refuses a job whose columns use such a set, before any row is read.
The session character set
Every connection is set to utf8mb4, collated utf8mb4_general_ci, after authentication, so a server’s init_connect setting cannot narrow it. Each source and destination connection then proves its session settings when it opens (SESSION_CHARSET_CHECK
). Comparisons in a job’s where and in GoArchive’s key SQL use that collation.
Columns are checked
Before any row is read, preflight refuses a job column the run reads or writes as text whose character set is not supported (COLUMN_CHARSET_CHECK ):
| Job type | Columns checked |
|---|---|
archive, copy-only | every column of every job table on the source, and every destination column of the same name |
purge | the primary-key and foreign_key columns of every job table, on the source |
ENUM and SET columns are checked. Numbers, temporal types, binary strings and JSON have no character set and are not. There is no bypass: both checks are enforced when preflight is bypassed, because the loss would be silent and could not be found after the run.
Supported character sets
| Character set | Note |
|---|---|
utf8mb4 | The session’s own character set |
utf8mb3 | |
ascii | Bytes 0x80β0xFF cannot be stored, so none reaches a run |
latin1 | |
latin2 | |
latin5 | |
latin7 | |
cp850 | |
cp852 | |
cp866 | |
keybcs2 | |
koi8r | |
koi8u | |
macce | |
macroman | |
utf16 | Every Unicode character, surrogate pairs included |
utf16le | Every Unicode character, surrogate pairs included |
utf32 | Every Unicode character |
binary | Listed for completeness: a binary string has no character set |
Refused character sets
Every other character set is refused:
| Character set | Why it is refused |
|---|---|
sjis | Two characters read as one; some codes read as ? |
cp932 | Two characters read as one (the NEC and IBM duplicates); some codes read as ? |
eucjpms | Two characters read as one (the NEC and IBM duplicates); some codes read as ? |
ujis | Two characters read as one (a full-width backslash and tilde); some codes read as ? |
armscii8 | Six punctuation bytes read as their ASCII look-alikes |
tis620 | Some codes read as U+FFFD |
big5 | Some codes read as ? or U+FFFD |
euckr | Some codes read as ? |
gb2312 | Some codes read as ? |
gbk | Some codes read as ? |
gb18030 | Some codes read as ? |
cp1250 | Unmapped bytes read as ? |
cp1251 | Unmapped bytes read as ? |
cp1256 | Unmapped bytes read as ? |
cp1257 | Unmapped bytes read as ? |
dec8 | Unmapped bytes read as ? |
geostd8 | Unmapped bytes read as ? |
greek | Unmapped bytes read as ? |
hebrew | Unmapped bytes read as ? |
hp8 | Unmapped bytes read as ? |
swe7 | Unmapped bytes read as ? |
ucs2 | Lone surrogates read as bytes that are not UTF-8 and cannot be bound back |
A refused column is converted to utf8mb4 on both servers, or its table is left out of the job.
