]> dreyeck.freedombox.rocks Git - topic-maps.git/blob - docs/slice-20h-iso-8859-5.md
Document qualified disposition of remaining XTM corpus cases
[topic-maps.git] / docs / slice-20h-iso-8859-5.md
1 # Slice 20H: ISO-8859-5
2
3 Base: 7f0e40d1558255773357c5ed4fa44bb5184d85e0 (Slice 20G).
4 This slice adds ISO-8859-5 byte decoding to the existing CTM path.
5
6 The existing SBCL 2.5.4 external format was qualified before implementation:
7 every byte from 00 through FF decodes without replacement, encodes back to the
8 original byte, and agrees with the independent local Python standard-library
9 codec. runtime-qualification.json preserves that result. Permanent Lisp tests
10 check the 256 decoded codepoints against identity mappings for 00–9F and an
11 explicit expected table for A0–FF. Production code uses SBCL's existing codec;
12 there are no downloaded tables or new runtime dependencies.
13
14 The normative basis is ISO/IEC 13250-6:2010 §§3.2 and 3.6, as published in the
15 [CTM specification](https://www.isotopicmaps.org/ctm/ctm.html).
16 Charset identities come from the
17 [IANA registry](https://www.iana.org/assignments/character-sets);
18 the expected high-byte table agrees with the
19 [Unicode ISO-8859-5 mapping](https://www.unicode.org/Public/MAPPINGS/ISO8859/8859-5.TXT).
20
21 ## Byte and prolog operations
22
23 ctm-encoding-kind recognizes only the two supported identities.
24 Names are case-insensitive; punctuation is significant.
25
26 | Identity | Accepted registered names |
27 | --- | --- |
28 | UTF-8 | UTF-8, csUTF8 |
29 | ISO-8859-5 | ISO-8859-5, ISO_8859-5:1988, ISO_8859-5, iso-ir-144, cyrillic, csISOLatinCyrillic |
30
31 compatible-ctm-encoding compares these identities, retaining the supplied names
32 in conflict diagnostics. decode-ctm-octets retains the selection order: external
33 argument, UTF-8 BOM, initial ASCII encoding declaration, conflict checks, UTF-8
34 default, then decoding. The strict UTF-8 decoder and IRI unescaping remain byte
35 identical to the base. The new ISO-8859-5 helper accepts specialized and generic
36 octet vectors without mutating them. read-prolog accepts the same supported
37 identity set after byte decoding. Unknown or unsupported charsets still signal
38 CTM-UNSUPPORTED / unsupported-encoding.
39
40 deserialize-file still reads binary data with an explicit Document IRI.
41 Include and CTM Mergemap still accept only explicitly resolved resource bytes;
42 the resource implementation, retrieval policy and identifier operations are
43 unchanged. Tests observe the actual byte-vector object reaching the shared
44 resource decoder. An externally supplied parent encoding is not inherited by a
45 resource with no declaration.
46
47 ## Original and controls
48
49 The existing Case Record for ctm/in/encoding-8859-5.ctm remains the registration
50 unit. The test checks all 42 original bytes and the pinned Document IRI.
51 The name bytes C0 DE E1 E1 D8 EF decode to Россия.
52 The original is actually read, checked semantically, canonicalized and compared
53 byte for byte before PASS is recorded. The expected byte count is read from the
54 pinned baseline. Topic handles, identifiers and map state remain unchanged by
55 canonicalization. A Unicode-string control produces the same baseline.
56
57 Permanent tests cover the six ISO names against all compatible external
58 arguments, mixed case, external-only input, ASCII, every A0–FF character in a
59 Name, all combinations of the two UTF-8 aliases with/without a BOM, and already
60 decoded Unicode strings. Include and Mergemap check both byte-vector forms,
61 child/parent Document IRIs and their different existing wildcard-counter rules.
62
63 The negative matrix checks conflicting declarations/BOM/external arguments,
64 unsupported ISO-8859-1, unknown and unregistered names, strict UTF-8 default,
65 misplaced/repeated/unquoted declarations, invalid versions, and child-resource
66 diagnostic context. Decoder spies prove conflicts and unsupported selections
67 fail before either codec is called.
68
69 Positions remain explicit: encoding preflight and UTF-8 failures use zero-based
70 byte offsets with no character line/column; lexer/prolog errors use zero-based
71 Unicode character offsets and one-based line/column. Paired ISO/UTF-8 fixtures
72 with Cyrillic text before an illegal character preserve the original byte
73 offset separately; only the ISO single-byte offsets coincide. Examples include
74 encoding-conflict at byte 14 (BOM/declaration), byte 0 (external/BOM), and byte
75 11 (external/declaration); invalid-utf8 at byte 10 for undeclared ISO input;
76 misplaced-encoding at character 23 for the repeated declaration.
77 current.encoding.json records every actual Condition, code, source, position,
78 unit and resource context.
79
80 Seven old Unsupported assertions now use still-unsupported ISO-8859-1 in newly
81 generated reports. They retain their Condition class, code and positions.
82 The validator permits only those exact fixture/message changes and one renamed
83 check label when comparing previous report contents; archived bytes remain
84 untouched. The original negative BOM-conflict contract still uses ISO-8859-1
85 and is unchanged.
86
87 ## Reproduction and acceptance
88
89 Run the existing complete offline driver, with an unused report output directory:
90
91 python3 scripts/validate-slice-20h.py \
92 --sbcl /nix/store/hlsiblxpxr1hax3dw9x8znd6zy097ddj-sbcl-2.5.4/bin/sbcl \
93 --output reports/slice-20h-rerun
94
95 The validator reuses pinned local XML sources, existing patches and the existing
96 test harness. Fresh SBCL processes disable user/system init files; XML, XTM and
97 the full regression execute under sandbox-exec with deny network*.
98 It checks retained TMDM/CTM/CXTM, Include, Mergemap, XTM identity/Mergemap,
99 Slices 20A–20G, Python tests and the NFC oracle. The encoding ASDF test system is
100 actually called by scripts/fresh-sbcl.lisp; its report is written only after
101 the tests finish.
102
103 Acceptance requires only this one NOT-RUN-to-PASS transition:
104 CTM 261 PASS / 4 FAIL / 8 NOT-RUN to 262 PASS / 4 FAIL / 7 NOT-RUN.
105 The separate XTM corpus must remain 6 PASS / 0 FAIL / 118 NOT-RUN.
106 All four existing CTM failure records and their actual CXTM must remain exact.
107 The validator verifies every pinned corpus size/hash, every prior report hash,
108 and all protected base files. It saves complete logs, stage reports,
109 actual/encoding-8859-5.ctm.cxtm, integrity hashes and execution.json.
110 It does not rebuild a Nix closure or load persistent Lisp images.
111
112 The managed sandbox initially fails before command execution because its
113 configured writable root /Users/rgb/.nix-profile is a symlink:
114 “writable root /Users/rgb/.nix-profile contains symlink component
115 /Users/rgb/.nix-profile; symlinked writable roots are not supported.”
116 Execution uses regular approved require_escalated calls with canonical Store
117 runtime paths. The symlink and sandbox configuration are unchanged; regression
118 subprocesses enforce the explicit network denial.