]> dreyeck.freedombox.rocks Git - topic-maps.git/blob - docs/slice-19d0b-cxml-encoding.md
Document normative conformance decisions for Slice 22C
[topic-maps.git] / docs / slice-19d0b-cxml-encoding.md
1 # Slice 19D0B — CXML encoding qualification
2
3 Base: `ec3f21e45c46849a6bb674194b6f7cc75b67bba1`.
4 Nixpkgs remains `50ab793786d9de88ee30ec4e4c24fb4236fc2674`.
5 Qualification is **PASS** with the local patches and mandatory byte boundary.
6 No XTM deserializer or XTM contract is introduced.
7
8 ## Normative decisions
9
10 The checked sources are [XML 1.0 Fifth Edition §2.2, §2.11, §4.3.3](https://www.w3.org/TR/2008/REC-xml-20081126/),
11 [Unicode 16.0 §3.9.2–3, Table 3-7 and §3.10](https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/),
12 [RFC 2781 §2.2, §3.3, §4](https://www.rfc-editor.org/rfc/rfc2781.html), and
13 [RFC 3629 §3–4](https://www.rfc-editor.org/rfc/rfc3629.html).
14
15 XML requires UTF-8 and UTF-16 support and accepts supplementary characters in
16 its Char range, including currently unassigned code points. Generic UTF-16
17 requires a BOM. UTF-16LE/BE are distinct encoding schemes: these original inputs
18 have an XML declaration that identifies their byte order. They are sufficiently
19 labeled. XML Appendix F.1 describes provisional signature detection; it is
20 informative, while declaration consistency and illegal-sequence rejection in
21 §4.3.3 are normative. Missing labels and contradictory declarations are errors.
22 XML line-end normalization also applies to UTF-16.
23
24 Unicode UTF-16 pairs a high surrogate with a low surrogate; isolated units are
25 invalid. UTF-8 must use its shortest form, valid continuation bytes and scalar
26 range, excluding encoded surrogates. Incomplete sequences require more input
27 until EOF, then fail. RFC 2781 distinguishes the explicit endian labels from
28 generic UTF-16; RFC 3629 corroborates the UTF-8 byte ranges. These encoding
29 rules do not change the project's Unicode NFC tables or add a Unicode library.
30
31 ## All eight original failures
32
33 The exact Unicode source text, **complete input octets in hex**, byte count,
34 SHA-256, detected encoding, before/after result, condition, diagnostic and
35 applicable clauses for each case are in
36 `reports/slice-19d0b/original-failures.json`. The source bytes are reconstructed
37 independently with Python's standard codecs and passed directly to CXML.
38 Diagnostics are observed parser messages; line/column reports are not promoted
39 to exact byte offsets.
40
41 | Original test | Bytes | Originally detected | Original result | Required/patched result | Cause |
42 |---|---:|---|---|---|---|
43 | `utf16le-bom:astral` | 114 | UTF-16 LE | rejects D83E DD8A | accepts `Grüße 水 🦊` | A |
44 | `utf16be-bom:astral` | 114 | UTF-16 BE | rejects D83E DD8A | accepts `Grüße 水 🦊` | A |
45 | `utf16le-explicit-no-bom:bmp` | 110 | UTF-8 | malformed markup | accepts `Grüße 水` | B |
46 | `utf16le-explicit-no-bom:astral` | 116 | UTF-8 | malformed markup | accepts `Grüße 水 🦊` | B + A |
47 | `utf16be-explicit-no-bom:bmp` | 110 | UTF-8 | NUL in markup | accepts `Grüße 水` | B |
48 | `utf16be-explicit-no-bom:astral` | 116 | UTF-8 | NUL in markup | accepts `Grüße 水 🦊` | B + A |
49 | `utf16-bom-without-declaration:astral` | 22 | UTF-16 LE | rejects D83E DD8A | accepts `水🦊` | A |
50 | `invalid-utf8` | 9 | UTF-8 | accepts `C3 28` as U+00E8 | rejects invalid continuation | C |
51
52 **A — closure-common defect.** The endian-specific `decode-sequence` methods
53 in `encodings.lisp` check the first code unit `x` again where they must check
54 the second, `y` (original lines 172/209). Valid supplementary content therefore
55 fails. Fixing `x` to `y` preserves the existing code-point computation.
56
57 **B — two implementation gaps, not incorrect expectations.** In
58 `closure-common/xstream.lisp`, `figure-encoding` recognizes only BOM-based
59 UTF-16. It defaults the four correctly labeled inputs to UTF-8 before CXML can
60 read their declarations. CXML's `setup-encoding` additionally lacks explicit
61 LE/BE handling, warns for generic UTF-16 and does not fully check consistency.
62 The patch recognizes the four-byte XML signatures provisionally, preserves
63 preread bytes and records the detection reason in the existing stream plist.
64 `setup-encoding` then checks the actual declaration. A signature alone does not
65 authorize generic BOM-less UTF-16, a missing label or a mismatched label.
66 The existing `set-full-speed` transition also checks that a declaration was
67 actually processed, so a leading ordinary processing instruction cannot bypass it.
68
69 **C — closure-common defect.** The UTF-8 `decode-sequence` method masks tail
70 bytes without validating their prefix. The patch validates continuation bytes,
71 shortest forms, the maximum scalar and permitted leading bytes; obsolete
72 five/six-byte forms fail. Incomplete UTF-8/UTF-16 sequences fail at EOF rather
73 than being silently dropped or repeatedly retried. Both existing UTF-16
74 decoders also receive their protocol-required CR/CRLF handling.
75
76 The four endian-specific, BOM-less inputs are valid supported forms, though
77 XML's mandatory UTF-8/UTF-16 support alone does not require these related
78 schemes. **No original expectation was weakened or replaced.** In particular,
79 the BOM-less tests were not changed to expect rejection. Their data includes
80 `3c 00 3f 00` / `00 3c 00 3f` followed by the appropriate encoding declaration.
81
82 ## Patch and source provenance
83
84 `patches/closure-common-19d0b.patch` changes only `encodings.lisp` and
85 `xstream.lisp`. `patches/cxml-19d0b.patch` changes only `xml/xml-parse.lisp`.
86 They modify the existing decoder and declaration paths; they do not introduce
87 a second parser, decoder framework or Unicode data set.
88
89 The original archives, seven Nix source directories and their recursive
90 source hashes remain unchanged. `flake.lock` and the corpus remain unchanged.
91 The Nix source environment now also records the local patch profile and
92 SHA-256 of both patch files. No further dependency was downloaded.
93
94 `scripts/prepare-cxml.py`:
95
96 1. Evaluates the existing pinned Nix source environment offline.
97 2. Verifies all seven original source hashes before writing outputs.
98 3. Copies only the two affected packages into a **new** caller-selected directory.
99 4. Checks local patch hashes against the Nix manifest and applies them with
100 the existing `patch` tool, with zero fuzz and no offsets.
101 5. Checks the exact set of changed files; records original and resulting
102 file hashes and the effective ASDF registry in `provenance.json`.
103
104 It never writes into `/nix/store`, overwrites an existing output directory or
105 retrieves a resource. Two independent copies produce identical patched file
106 hashes. Effective paths in recorded validation runs belonged to those temporary
107 execution contexts; they are not persistent addresses. Reproduce them with the
108 preparation command rather than reusing expired temporary paths.
109
110 ## Binding public call contract
111
112 The optional ASDF system **`topic-maps/xml`** depends only on CXML, not on CTM,
113 TMDM or HyperDoc. Future XTM deserialization **must call
114 `topic-maps.xml:parse-octets`**, not bare `cxml:parse-octets`.
115
116 The function accepts a specialized octet vector and a SAX handler, and returns
117 the handler's SAX result without replacing its object identity. It:
118
119 * Refuses libraries without the required local patch ABI markers.
120 * Fixes namespace processing on and gives the caller no resolver/DTD overrides.
121 * Wraps the receiver in an existing CXML SAX proxy that rejects every DOCTYPE
122 before forwarding it. Even a NIL/permissive receiver cannot bypass the guard.
123 * Disallows internal subsets and uses a resolver that **signals**
124 `external-resource-forbidden`; returning NIL is forbidden.
125 * Disables catalog lookup and binds a fresh DTD cache for each parse.
126
127 The ABI markers protect against accidental loading of pristine dependencies;
128 they are not an integrity defense against arbitrary Lisp code rewriting loaded
129 functions. Source and patch integrity is checked during preparation.
130 The underlying library's unsafe entry points are not made globally safe.
131 The public boundary offers no already-decoded-string, filename, transport
132 encoding or arbitrary parser-option interface.
133
134 ## Reproduction and inspection
135
136 ```sh
137 cd /Users/rgb/workspace/topic-maps
138 xml_sources=$(mktemp -d /tmp/topic-maps-xml.XXXXXX)
139 export CL_SOURCE_REGISTRY="$(python3 scripts/prepare-cxml.py \
140 --directory "$xml_sources/patched" --registry)"
141 /usr/bin/sandbox-exec -p '(version 1)(allow default)(deny network*)' \
142 /nix/store/hlsiblxpxr1hax3dw9x8znd6zy097ddj-sbcl-2.5.4/bin/sbcl \
143 --noinform --no-sysinit --no-userinit --script tests/cxml-probe.lisp
144
145 python3 scripts/validate-cxml-19d0b.py \
146 --sbcl /nix/store/hlsiblxpxr1hax3dw9x8znd6zy097ddj-sbcl-2.5.4/bin/sbcl
147 ```
148
149 In a fresh Lisp process with that registry:
150
151 ```lisp
152 (require :asdf)
153 (asdf:load-asd (truename "topic-maps.asd"))
154 (asdf:load-system "topic-maps/xml")
155 (let ((bytes (make-array 4 :element-type '(unsigned-byte 8)
156 :initial-contents '(60 114 47 62)))) ; <r/>
157 (describe (topic-maps.xml:parse-octets bytes (cxml-xmls:make-xmls-builder))))
158 ```
159
160 The Nix manifest's `sourceRegistry` still denotes pristine upstream sources;
161 the preparation step is mandatory for this API. Merely entering the old
162 DevShell or loading unpatched CXML does not activate the patches. A fresh-process
163 counterprobe loads precisely that unpatched environment and proves that the
164 public boundary refuses it before parsing. This source-overlay workflow reuses
165 the existing Nix/ASDF provisioning without realizing an uncached toolchain.
166
167 ## Validation evidence
168
169 `reports/slice-19d0b/execution.json` records source/patch hashes, runtime and
170 timestamps, independent source-copy reproduction, original and patched runs,
171 the fresh unpatched-boundary counterprobe and the complete retained regression.
172
173 * Original test program from the base commit, pristine sources: **50 PASS, 8 FAIL**.
174 * **Unmodified original program**, patched sources: **58 PASS, 0 FAIL**.
175 * Original 58 expectations through the binding public boundary, plus 88 focused
176 regressions: **146 PASS, 0 FAIL**, repeated in two fresh SBCL 2.5.4 processes.
177 * Additional cases cover malformed/overlong/truncated UTF-8, invalid UTF-16
178 units, Unicode scalar boundaries, partial-input/EOF behavior, large refills,
179 line ends, missing/conflicting declarations, namespaces and the fixed API.
180 * All external DTD/entity rejection checks and the NIL-resolver fallback
181 counterprobe remain. The fallback is intercepted before OPEN, so sandbox I/O
182 denial is never misreported as successful resolver refusal.
183 * Full existing regression: **12,100 Lisp checks**, **2 Python tests** and
184 **16,294 NFC oracle comparisons** pass. All 24 retained JSON reports are
185 byte-identical to Slice 19C. CTM stays **204 PASS, 4 known FAIL, 65 NOT-RUN**.
186
187 All Lisp observations run with network access denied. Nix evaluation uses
188 `--offline`; patching consumes already materialized sources. The complete Nix
189 toolchain/DevShell offline rebuild remains **NOT-RUN**. This is an offline
190 source-patch/ASDF qualification using existing Nix runtime artifacts, not a claim
191 that every Nix artifact was rebuilt offline or that all of XML is conformant.
192 No XTM PASS, push, deployment, HyperDoc modification or persistent-image load.