]> dreyeck.freedombox.rocks Git - topic-maps.git/blob - docs/slice-19d0-cxml.md
Verify CTM/XTM semantic equivalence through TMDM and CXTM (Slice 22F)
[topic-maps.git] / docs / slice-19d0-cxml.md
1 # Slice 19D0 — Pinned CXML provisioning and qualification
2
3 Source provisioning succeeds. **Unmodified parser qualification fails.**
4 This is not an XTM implementation, and it does not authorize proceeding to 19D1.
5
6 ## Source identity
7
8 Base: `bc4fcf0f126bab6567b37a21fc8c8eab2944c152`.
9 The unchanged Nixpkgs pin is `50ab793786d9de88ee30ec4e4c24fb4236fc2674`.
10 `lib.cxmlEnvironment` follows the pinned CXML package's transitive `lispLibs`
11 and uses their **sources**, not Nixpkgs SBCL 2.4.10 FASLs.
12
13 All hashes below are recursive SHA-256 NAR hashes in Nix base32, not compressed
14 archive hashes. They come from the pinned package definitions and are checked
15 again against the materialized source directories by `validate-cxml.py`.
16
17 | Package | Version | SHA-256 |
18 |---|---|---|
19 | cxml | 20200610-git | `18fls3bx7vmnxfa6qara8fxp316d8kb3izar0kysvqg6l0a45a51` |
20 | closure-common | 20181018-git | `0k5r2qxn122pxi301ijir3nayi9sg4d7yiy276l36qmzwhp4mg5n` |
21 | babel | 20241012-git | `0359bj3yr6frybcmg8qr5vi4q8hzbsb7hmvxdc0jgkfz3c33q667` |
22 | alexandria | 20241012-git | `0jq0n59s0r9yl374f0zpdnaflb5g853yqvax7ka8rnypspyykwdw` |
23 | trivial-features | 20230618-git | `0r33ycg1wsmglbsychglzkd6fachnnqfzd0w9mhpwi6cz94hx7c3` |
24 | trivial-gray-streams | 20241012-git | `0iw6q5hx7x8sc5s7ikvsjccsksbm0rd13d54mkrg62sc56hjywrm` |
25 | puri | 20201016-git | `0gq2rsr0aihs0z20v4zqvmdl4szq53b52rh97pvnmwrlbn4mapmd` |
26
27 Dependency graph: CXML → closure-common, PURI, trivial-gray-streams;
28 closure-common → Babel, trivial-gray-streams; Babel → Alexandria, trivial-features.
29 Only CXML and closure-common required the separately authorized acquisition:
30
31 ```sh
32 nix store prefetch-file --json --unpack --name source \
33 --expected-hash sha256-oahCFKDm4a39BFn9ONZEzYRxu0MqK2yU67bu09fQ1KE= \
34 http://beta.quicklisp.org/archive/cxml/2020-06-10/cxml-20200610-git.tgz
35 nix store prefetch-file --json --unpack --name source \
36 --expected-hash sha256-trxKLuS/YjOoOcJHfxp5OkWv7MhRxgBG7FeIYDsWuUw= \
37 http://beta.quicklisp.org/archive/closure-common/2018-10-18/closure-common-20181018-git.tgz
38 ```
39
40 Both returned exactly the expected hashes and source paths. The other five
41 source directories already existed. No other package was fetched; there is
42 no Quicklisp installation, runtime download, upstream patch or new Nixpkgs pin.
43
44 ## Offline loading and reproduction
45
46 The default DevShell exports the same `CL_SOURCE_REGISTRY` as the independently
47 evaluable source environment. It excludes inherited user registries; the project
48 itself can still be loaded explicitly with `asdf:load-asd` as before.
49
50 ```sh
51 cd /Users/rgb/workspace/topic-maps
52 export CL_SOURCE_REGISTRY="$(nix eval --offline --option eval-cache false --raw \
53 .#lib.cxmlEnvironment --apply 'f: (f "aarch64-darwin").sourceRegistry')"
54 /usr/bin/sandbox-exec -p '(version 1)(allow default)(deny network*)' \
55 /nix/store/hlsiblxpxr1hax3dw9x8znd6zy097ddj-sbcl-2.5.4/bin/sbcl \
56 --noinform --no-sysinit --no-userinit --non-interactive \
57 --eval '(require :asdf)' --eval '(asdf:load-system "cxml")'
58
59 python3 scripts/validate-cxml.py \
60 --sbcl /nix/store/hlsiblxpxr1hax3dw9x8znd6zy097ddj-sbcl-2.5.4/bin/sbcl
61 ```
62
63 The first command loads CXML successfully. The validator deliberately exits
64 **1** when XML qualification fails, after saving both XML runs and the complete
65 retained regression. It never converts a failed qualification to PASS merely
66 because source provisioning or CTM regression succeeds.
67
68 The per-command `eval-cache false` avoids a local Nix evaluation-cache error
69 about `dynamic-derivations`; no Nix settings or cache contents are changed.
70 The Lisp runs receive the evaluated Nix source environment and use the existing
71 Nix-store SBCL 2.5.4. They do not run inside a newly built default DevShell.
72 The pinned default DevShell/toolchain is not fully provisioned: building it and
73 a complete offline rebuild are **NOT-RUN**, not implied by these tests.
74 No FASL cache policy, relocation or second SBCL version is introduced.
75
76 ## Explicit external-resource policy
77
78 `tests/cxml-probe.lisp` contains a qualification-only SAX handler and parsing
79 configuration. It is not a production XML or XTM API and is not installed
80 globally by ASDF loading. Future callers must explicitly apply the policy:
81
82 * Raw octet input, namespace processing enabled, no DTD validation.
83 * Every DOCTYPE is rejected by `sax:start-dtd`, before subset processing.
84 * `:disallow-internal-subset t` is a second guard against internal subsets.
85 * `:entity-resolver` **signals** `external-resource-forbidden`; it never returns
86 NIL. Catalog resolution is disabled and each parse has a fresh DTD cache.
87
88 The tests exercise local-file and HTTP external DTDs, general entities and
89 parameter entities. Resolver-only probes temporarily permit DOCTYPEs so they
90 test the resolver independently of the first guard. A test-only interception
91 of `cxml::uri-to-pathname` traps the exact fallback before OPEN; it is restored
92 in the disposable process. The rejecting resolver never reaches this fallback.
93 A NIL-resolver counterprobe does reach it and is intercepted before any I/O.
94 Thus filesystem/network denial is not mistaken for successful resolver refusal.
95 The complete probes run twice in fresh processes with all network access denied.
96
97 CXML documents the NIL fallback in `xml/xml-parse.lisp:3179`; the fallback OPEN
98 is at line 1064, and the DOCTYPE callback precedes subset processing at line 2578
99 in the unchanged CXML source. No external resource is fetched by these probes.
100
101 ## Observed encoding defects — not normalized away
102
103 The repeated matrix records **50 PASS and 8 FAIL per process**. Namespaces,
104 expanded attribute names, duplicate-attribute rejection, nesting errors,
105 ordinary UTF-8, BMP UTF-16 with BOM, XML line/attribute normalization and the
106 external-resource policy pass. All eight failures remain visible:
107
108 1. **Valid UTF-16 supplementary characters rejected:** U+1F98A has surrogate
109 code units D83E DD8A (LE bytes `3e d8 8a dd`). The LE/BE BOM inputs are
110 rejected, as is the LE BOM input without an XML declaration. In the original
111 `closure-common/encodings.lisp:172` and `:209`, the second-code-unit range
112 check examines `x` again instead of `y`. These are three failed matrix cells.
113 2. **Invalid UTF-8 accepted:** the content bytes `c3 28` are not valid UTF-8.
114 CXML accepts them and reports U+00E8. The UTF-8 decoder beginning at
115 `closure-common/encodings.lisp:222` masks continuation bytes without checking
116 their required prefix. This is one failed rejection test. Python's strict
117 codecs independently confirm both this invalid sequence and the valid
118 UTF-16 surrogate pair; they are byte-fixture checks, not a substitute parser.
119 3. **Additional encoding-support observations:** explicit `UTF-16LE`/`UTF-16BE`
120 inputs without BOM fail for BMP and supplementary content (four cells).
121 These additional encoding-name forms are reported separately from the
122 decisive XML UTF-8/UTF-16 failures above. The pinned input stream recognizes
123 UTF-16 by BOM only (`closure-common/xstream.lisp:366`). Generic `UTF-16`
124 declarations also produce an unsupported-encoding warning even when BOM
125 detection lets BMP input parse; `cxml/xml/xml-parse.lisp:2520` is involved.
126
127 The original libraries are not repaired, pre-decoded, filtered or replaced in
128 these tests. A dependency correction needs an explicit, reviewable follow-up:
129 repair/replace the encoding path with pinned, tested changes, retain upstream
130 source hashes, and re-run strict malformed-input and surrogate-boundary tests.
131 This is more than making the already pinned sources available. No new parser
132 implementation, unapproved source acquisition or XTM work was started.
133
134 ## Retained regression and evidence
135
136 `reports/slice-19d0/execution.json` records the pinned manifest, actual NAR
137 hashes, source environment, both fresh XML runs, failed checks and observation
138 times. `cxml-run-1.log` and `cxml-run-2.log` contain conditions and SAX checks.
139 `regression-execution.json` and `regression-tests.log` preserve the complete
140 unchanged regression run. Every retained JSON report is compared byte-for-byte
141 with Slice 19C; original sources, baselines and normative references stay intact.
142
143 Retained results: **12,100 Lisp checks**, **2 Python tests**, **16,294 NFC oracle
144 comparisons** pass. CTM remains **204 PASS, 4 known FAIL, 65 NOT-RUN**.
145 No XTM contract is registered or marked PASS.
146
147 Conclusion: **source provisioning PASS; parser qualification FAIL; full offline
148 DevShell rebuild NOT-RUN.** The parser is not yet qualified for Slice 19D1.