]> dreyeck.freedombox.rocks Git - topic-maps.git/blob - docs/slice-7-validation.md
feat(ctm): decode ISO-8859-5 with qualified charset aliases
[topic-maps.git] / docs / slice-7-validation.md
1 # Slice 7 — Integer Occurrences: validation evidence
2
3 Repository: `/Users/rgb/workspace/topic-maps`.
4 Observed initial branch: `codex/ctm-slice-6`, clean working tree, HEAD
5 `3924d1eac5a070fe5a6a26683e23ac338793f544` (the requested base).
6 Created `codex/ctm-slice-7` from that verified commit. Corpus and reference files
7 remain unchanged at pin `cd165b150176c1d6842b37be9cc9051b40d4996b`.
8
9 ## Normative interpretation and implemented boundary
10
11 Consulted the published [CTM §3.4.1](https://www.isotopicmaps.org/ctm/ctm.html),
12 [TMDM §§4.3–4.4, 5.6, 6.5](https://www.isotopicmaps.org/sam/sam-model/2006-06-18/),
13 [CXTM §§3.16, 4.2–4.3, 4.9](https://www.isotopicmaps.org/cxtm/cxtm.html) and
14 [XML Schema Part 2 §§3.3.13.1–3.3.13.2, 4.3.6](https://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#integer).
15 The full interpretation, original sources, actual object graph and inspection
16 example are in `reading-integer-occurrences.md`.
17
18 Three different contracts are retained:
19
20 1. The reader represents an **unquoted integer value** with a canonical lexical
21 string. Its lexer token and original CTM source retain the source spelling.
22 This early representation is a documented implementation policy under CTM's
23 value mapping, not a claim that CTM explicitly requires early canonicalization.
24 A qualified **quoted string** retains its escape-decoded value instead.
25 2. TMDM stores a string plus a separate datatype locator. Its API does not
26 normalize integer strings. Occurrence equality/merging uses stored string
27 equality and the other §5.6 properties; numeric equivalence or equal XML
28 does not replace those rules.
29 3. CXTM formats xsd:integer output canonically, without changing those model
30 values or retained merge handles. Occurrence sorting compares the original
31 stored strings (§§4.3/4.9), not numeric values or reformatted output strings.
32
33 Canonical formatting is character-based, removing plus signs/leading zeros and
34 using `0` for positive/negative zero. It has no machine-word overflow boundary.
35 XML whitespace at the ends of a qualified integer is accepted for XSD datatype
36 interpretation, retained in TMDM, and removed in canonical output. Internal
37 whitespace and non-ASCII decimal digits are rejected for xsd:integer.
38
39 This explicitly distinguishes cases where bare `001`/`+1`/`1` already have the
40 same stored string from quoted `"001"`/`"1"`, whose strings remain different.
41 The latter produce equal canonical value elements but remain separate items.
42 A dedicated test also shows two separate maps with equal CXTM and different
43 stored integer strings. No XML-based equality or generic numeric merging exists.
44
45 ## Selected original CTM/CXTM contracts
46
47 All sources and original baselines were read before implementation. All six
48 use the existing deterministic absolute Document IRI, also as CXTM Base Locator.
49 Actual UTF-8 bytes are compared directly against original baseline bytes, with
50 no trimming, XML reparsing, baseline rewriting or numeric preprocessing.
51
52 All cases produce one Topic Map, no Names/Associations and no datatype Topic.
53 Local Topic identities are only `DocumentIRI#identifier` Item Identifiers;
54 SI/SL sets and Occurrence Item Identifiers are empty. The datatype is always
55 `http://www.w3.org/2001/XMLSchema#integer`, never a Type Topic.
56
57 | Original source | Actual stored values, source order | Items/references | Byte-exact result |
58 | --- | --- | --- | --- |
59 | `ctm/in/occurrence-integer.ctm` | `"1"` | 2 Topics, 1 Occurrence; Parent EQ topic, Type EQ occtype | PASS, 351 bytes |
60 | `ctm/in/occurrence-integer2.ctm` | `"1"` | same; `1.` leaves the topic terminator | PASS, 351 bytes |
61 | `ctm/in/occurrence-integer-plus-sign.ctm` | `"1"` | same; original `+1` | PASS, 351 bytes |
62 | `ctm/in/occurrence-integer-minus-sign.ctm` | `"-1"` | same | PASS, 352 bytes |
63 | `ctm/in/occurrence-integer-leading-zeros.ctm` | `"1", "1", "-1", "100", "0", "0", "0"` | 8 Topics, 7 Occurrences; same Parent, Types EQ o1…o7 | PASS, 1,741 bytes |
64 | `ctm/in/occurrence-explicit-datatype2.ctm` | `"42"` | 2 Topics, 1 Occurrence; Parent EQ topic, Type EQ occtype | PASS, 352 bytes |
65
66 The leading-zero case's canonical occurrence order is o3, o5, o6, o7, o1, o2,
67 o4. The equal value strings are separated by Type, with canonical Type Topics
68 1–7 and owner Topic 8. The five simpler maps use Type Topic 1 and owner Topic 2.
69
70 | Imported contracts | PASS | FAIL | UNSUPPORTED | NOT-RUN |
71 | --- | ---: | ---: | ---: | ---: |
72 | All 273 cases | 42 | 0 | 0 | 231 |
73 | Positive CXTM contracts | 39 | 0 | 0 | 192 |
74 | Negative CTM contracts | 3 | 0 | 0 | 39 |
75
76 Only the six selected contracts changed NOT-RUN → PASS. The previous 33 positive
77 case records/outputs and all three specific prefix rejections are unchanged.
78 The existing identity, merge, Association, Occurrence and prefix object-observation
79 records match Slice 6 exactly. No unexecuted imported contract is inferred to
80 pass or to be unsupported from other tests.
81
82 ## Fresh offline regression evidence
83
84 Each suite ran again in a fresh SBCL process, ignoring user/system initialization
85 and inherited ASDF source configuration. `sandbox-exec` denied network access.
86 The 2.4.10 run used an independently copied checkout under `/private/tmp`.
87
88 | Suite | SBCL 2.5.4 PASS | SBCL 2.4.10, relocated PASS |
89 | --- | ---: | ---: |
90 | Foundation | 54 | 54 |
91 | Topic Identity | 123 | 123 |
92 | Topic Merging | 646 | 646 |
93 | Associations/Roles | 657 | 657 |
94 | Retained CXTM | 445 | 445 |
95 | Occurrences | 1,101 | 1,101 |
96 | Prefix/explicit datatypes | 150 | 150 |
97 | Integer checks | 665 | 665 |
98 | Total Lisp checks | 3,841 | 3,841 |
99 | Python infrastructure tests | 2 | 2 |
100 | Independent NFC oracle comparisons | 16,294 | 16,294 |
101 | Independent integer arithmetic probes | 143 | 143 |
102 | Original positive CXTM byte comparisons | 39 | 39 |
103
104 Final failures: **0**. All eight report pairs are byte-identical across runtimes
105 and relocation. Public DESCRIBE/serialization inspection was independently
106 executed under network denial and passed. The 143 numeric probes include
107 positive/negative 10,000-digit integers and XML whitespace. Python arithmetic
108 provides an independent implementation of the numeric-to-string result.
109 The new integer tests also exercise malformed values, source diagnostics,
110 explicit/implicit datatypes, lexical equality, retained handles after Parent/Type
111 merging, datatype/type distinctions, ordering permutations and nonmutation.
112
113 Previously unsupported expectations were adapted one-for-one: the `42.`
114 unsupported fixture now uses excluded `42.5.`; the private unsupported integer
115 fixture now uses Decimal; the qualified integer CXTM assertion now requires
116 successful canonical output instead of an unsupported condition. Retained suite
117 totals remain 1,101 and 150. The original 445 canonical check labels/results are
118 unchanged. No original CTM source or expected XML was altered.
119
120 Evidence in `reports/slice-7/`: raw process logs, `execution.json` with commands,
121 runtime versions/paths, exit codes and artifact SHA-256 hashes; eight reports;
122 all 39 actual CXTM byte files; and `describe.log`. The combined CXTM report's
123 2,361 checks are retained CXTM + Occurrence + prefix + integer checks, not an
124 additional suite to add to the Lisp total. Development fixture syntax/layout
125 corrections precede these final successful runs.
126
127 ## Reproduction
128
129 With the installed runtimes and Python UCD 15.0/15.1:
130
131 ```sh
132 python3 /Users/rgb/workspace/topic-maps/scripts/validate-slice-7.py \
133 --sbcl-254 /nix/store/hlsiblxpxr1hax3dw9x8znd6zy097ddj-sbcl-2.5.4/bin/sbcl \
134 --sbcl-2410 /nix/store/y96nx8571mpdvjd40krkdim2av0in972-sbcl-2.4.10/bin/sbcl \
135 --output /private/tmp/topic-maps-slice-7-reproduction
136 ```
137
138 The script verifies runtime versions, applies OS network denial, runs both fresh
139 suites, relocates the second checkout, compares reports and previous records,
140 checks original baseline bytes, executes the documented inspection and records
141 the evidence. It provisions no software, changes no corpus and makes no commit.
142
143 Single fresh offline run:
144
145 ```sh
146 /usr/bin/sandbox-exec -p '(version 1)(allow default)(deny network*)' \
147 /usr/bin/env PYTHONDONTWRITEBYTECODE=1 \
148 SBCL=/nix/store/hlsiblxpxr1hax3dw9x8znd6zy097ddj-sbcl-2.5.4/bin/sbcl \
149 TOPIC_MAPS_REPORT=/private/tmp/topic-maps-slice-7.json \
150 /Users/rgb/workspace/topic-maps/scripts/test.sh
151 ```
152
153 ## Implementation and remaining limits
154
155 `src/ctm/lexer.lisp` contains the single numeric token path and the relocated
156 existing excluded-literal recognizer. `src/ctm/reader.lisp` maps integer token
157 values and qualified strings to the existing model. `src/tmdm/integers.lisp`
158 provides pure lexical helpers; `occurrences.lisp` validates integer input before
159 mutation without rewriting it. CXTM adds integer validation/output but keeps
160 its existing stored-string comparator. `tests/ctm-integers.lisp` and independent
161 Python checking cover the six selected contracts and direct semantics.
162
163 Decimal, Date, DateTime, Variants, nonempty Scope, Reification, Templates,
164 cross-document merging and HyperDoc integration remain excluded. No `*` literal
165 counter semantics or derived-integer subtype conversion was introduced; none
166 is required by the six selected cases. General locator/Unicode/root-LF limits
167 remain as documented in preceding slices. No server was started or deployed.
168
169 No blocking standard/corpus conflict was observed for this slice. The reader's
170 early bare-integer value-representation policy is explicitly documented, while
171 qualified string equality and normative string-based sort rules are retained.
172 Recommendation: **PASS for Slice 7's bounded scope**, await review before any
173 further slice.