]> dreyeck.freedombox.rocks Git - topic-maps.git/blob - README.md
Implement XTM association reification with native conformance evidence
[topic-maps.git] / README.md
1 # Common Lisp Topic Maps โ€” Slice 19D2
2
3 Slice 19D2 adds the explicit **`topic-maps/ctm/xtm`** adapter for `%mergemap`
4 with `http://psi.topicmaps.org/iso13250/xtm`. The existing resolver passes raw
5 bytes to the qualified XTM reader; existing TMDM Map-content import transfers
6 its result. The unchanged 1,519-byte original baseline matches end-to-end.
7 CTM now has **205 PASS / 4 known FAIL / 64 NOT-RUN** with the adapter loaded;
8 the separate XTM corpus has **6 PASS / 0 FAIL / 118 NOT-RUN**.
9 See [XTM Mergemap: API, objects, diagnostics and reproduction](docs/slice-19d2-xtm-mergemap.md).
10
11
12 Slice 19D1 adds **`topic-maps/xtm`**, an XTM 2.0 identity/`instanceOf` subset.
13 Six unchanged XTM sources match their original CXTM baselines byte-for-byte;
14 results are separate from the unchanged **204 PASS / 4 FAIL / 65 NOT-RUN** CTM
15 contracts. No CTM XTM-Mergemap integration or full XTM conformance is claimed.
16 See [the XTM API, model and reproduction guide](docs/slice-19d1-xtm-identity.md).
17
18
19 Slice 19D0B qualifies local, versioned CXML/closure-common encoding corrections:
20 **58 original and 88 additional XML checks PASS** in fresh offline SBCL 2.5.4.
21 The required entry point is `topic-maps.xml:parse-octets` after offline source
22 patch preparation; it enforces external-resource denial. Slice 19D1 uses that boundary for its independent XTM 2.0 identity reader. See [encoding qualification](docs/slice-19d0b-cxml-encoding.md).
23
24 Slice 19D0 made pinned CXML sources available to ASDF through the Nix source
25 environment. Its initial parser qualification failed on UTF-16 supplementary
26 characters and invalid UTF-8; those historical failures are retained. See
27 [CXML provisioning and qualification](docs/slice-19d0-cxml.md) for exact hashes,
28 offline loading, external-entity policy, the failing probes and retained regression.
29
30 Slice 19C adds `%include` with controlled resources, per-level Topic-II aliases,
31 immediate-parent Document/Wildcard environments, Counter propagation and
32 source-preserving Template import. Imported bodies execute in the caller's
33 identity/Prefix environment while errors preserve definition and invocation
34 sources. See [Reading CTM Include](docs/reading-ctm-include.md) and
35 [Slice 19C validation](docs/slice-19c-validation.md).
36 The registered corpus is **204 PASS, 4 FAIL, 65 NOT-RUN**. All twelve Include
37 contracts execute: seven byte-equal positives, three intended rejections, two
38 explicit normative/baseline mismatches. The two earlier DateTime FAILs remain.
39
40 Slice 19B adds CTM `%mergemap` with the CTM notation and an explicit
41 `:resource-resolver` on `deserialize-string` / `deserialize-file`. Resources are
42 supplied as bytes by the caller, deserialized with independent environments,
43 and transferred by Slice 19A's TMDM content import. No implicit file/network
44 retrieval is provided (Include is added by Slice 19C). XTM Mergemap is added by the explicit Slice 19D2 adapter.
45 See [Reading CTM Mergemap](docs/reading-ctm-mergemap.md) and
46 [Slice 19B validation](docs/slice-19b-validation.md).
47 The registered corpus now has **194 PASS, 2 known DateTime FAIL, 77 NOT-RUN**:
48 four new byte-exact positive contracts and three intended negative rejections.
49
50 Slice 19A adds `topic-maps.tmdm:import-map-content`: source-preserving copying
51 of Topics, Associations and their children into a target Map, followed by the
52 existing same-map identity and structural merging. Both original graphs remain
53 unchanged on semantic rejection. The returned resolver tracks canonical target
54 representatives. Source Map identifiers/reifier are excluded.
55 See [Reading Map Content Import](docs/reading-map-content-import.md) for the API,
56 object references and inspection forms, and [Slice 19A validation](docs/slice-19a-validation.md)
57 for fresh offline execution evidence. This adds no CTM import contracts:
58 **187 PASS, 2 known DateTime FAIL, 84 NOT-RUN** remain unchanged.
59
60 A working ANSI Common Lisp reader for CTM identity/name syntax and
61 Associations with one or more Roles, with same-map merging of Topics, Names, Associations
62 and Roles, plus string/IRI-valued Occurrences, in a bounded TMDM model. HyperDoc remains a possible future client
63 and is neither the codebase nor a dependency. Slice 4 adds a pure, independent
64 CXTM canonicalizer for this model. Controlled Include/Mergemap and Map content import are described above. Slice 5 adds Occurrence items and replacement merging
65 without expanding Scope, Reification or Variants. Slice 6 adds document-local
66 prefixes, QName resolution and explicit datatype locators on quoted Occurrence
67 values. Unknown datatypes retain their lexical values; Date timezone CXTM rules
68 remain explicitly unsupported. Slice 7 adds integer literals, lexical validation
69 and pure xsd:integer canonical output; qualified strings retain their stored
70 lexical values. Slice 8 adds decimal literals, exact character-based xsd:decimal
71 canonicalization and lexical validation, with the same separation of stored
72 strings, TMDM equality and canonical serialization. Slice 9 adds timezone-free
73 xsd:date literals, Gregorian calendar validation and pure CXTM Date output.
74 Negative-year leap validation follows the documented XSD 1.0 E.1 interpretation.
75 Slice 10 adds xsd:dateTime with absent/zero zones, exact fractional precision and
76 midnight rollover. A noncandidate leap-second baseline disagrees with XSD D.1
77 following-minute interpretation and remains FAIL. Slice 11 adds exact nonzero-offset UTC canonicalization; the original timezone2
78 contract now executes and reports its known inconsistent baseline as FAIL.
79 Nonzero-offset BCE day rollover and leap-second contexts remain explicitly
80 unsupported; existing zero/absent-zone behavior is retained.
81 Slice 12A adds explicitly typed Topic Names. Slice 13 adds Scope Topic sets to
82 Names, Occurrences and Associations, including reference replacement, duplicate
83 merging and canonical output. Slice 14 adds unreified Variants with proper Scope
84 supersets, canonical merge handles and pure CXTM output. Slice 15 adds per-document
85 Wildcards and recursive Embedded Topics through the shared Topic reference reader.
86 Slice 16A adds general Item Identifiers for all seven existing information-item
87 kinds, including metadata-preserving replacement merges and pure CXTM output.
88 Slice 16B adds same-map Reification, full merge cascades, atomic semantic
89 preflight and all six CTM Reifier attachment positions.
90 Slice 17A removes the two-Role CTM limit, preserving the existing model and
91 canonicalizer. Slice 17B adds `isa`/`ako` using ordinary Associations and protects
92 the distinct Core Subjects defined by TMDM 2006 ยง7. Atomic merge preflight uses
93 the existing disposable TMDM graph. No inference engine is introduced.
94
95 Slice 18 adds grammar-checked Templates, contextual positional arguments,
96 global/Tail/nested Invocations, invocation-local Wildcards and canonical handles
97 through existing model operations. Seven explicitly documented grammar contracts
98 remain open; no extra TMDM item kind or Lisp evaluation is introduced.
99
100 Start with [CTM Templates and Invocations](docs/reading-ctm-templates.md) for
101 representation, diagnostics, open grammar cases and executable inspection forms.
102 See [Slice 18 validation](docs/slice-18-validation.md) for the full offline run,
103 57 additional byte-exact original contracts and ten specific rejection contracts.
104
105 Start with [isa/ako and Core Subject Consistency](docs/reading-type-relationships.md)
106 for the exact vocabulary, object relations, normative distinctions and inspection.
107 See [Slice 17B validation](docs/slice-17b-validation.md) for the complete offline
108 regression and thirteen new original byte-exact comparisons.
109
110 Start with [Associations with Arbitrarily Many Roles](docs/reading-association-arity.md)
111 for the actual seven-Topic/three-Role example and executable inspection forms.
112 See [Slice 17A validation](docs/slice-17a-validation.md) for the full offline run
113 and the single new byte-exact original contract.
114
115 Start with [Reification](docs/reading-reification.md) for actual Reifier/Reified
116 objects, normative clauses, cascade handling and inspection forms.
117 See [General Item Identifiers](docs/reading-item-identifiers.md) for the
118 uniform API, retained handles, exact boundaries and executable inspection forms.
119
120 Systems: `topic-maps` (the retained foundation), `topic-maps/tests`,
121 `topic-maps/tmdm`, `topic-maps/ctm`, `topic-maps/ctm/tests` and
122 `topic-maps/ctm/merging-tests` and `topic-maps/ctm/association-tests`, plus `topic-maps/cxtm` and
123 `topic-maps/cxtm/tests`, `topic-maps/ctm/occurrence-tests` and `topic-maps/ctm/prefix-tests` and `topic-maps/ctm/integer-tests`, plus `topic-maps/ctm/decimal-tests` and `topic-maps/ctm/date-tests` and `topic-maps/ctm/datetime-tests` and `topic-maps/ctm/timezone-tests`, `topic-maps/ctm/typed-name-tests` and `topic-maps/ctm/scope-tests`, plus `topic-maps/ctm/variant-tests` and `topic-maps/ctm/reference-tests`.
124 The core reader/model uses ANSI Common Lisp; the foundation/tooling uses
125 ASDF/UIOP and Python's standard library. No Quicklisp packages are required.
126 `topic-maps/tmdm/identifier-tests` and `topic-maps/tmdm/reification-tests` add the
127 general identifier and Reification regression suites.
128 `topic-maps/ctm/association-arity-tests` adds n-ary syntax, merge/handle checks
129 and the original `association-duplicate-role.ctm` comparison.
130 `topic-maps/ctm/type-relationship-tests` adds Core consistency, atomic failure
131 and isa/ako model/reference-graph checks.
132
133 Start with [Reading Topic References](docs/reading-topic-references.md) for Wildcard
134 IRIs/counters, recursive Embedded Topics and actual object relationships.
135 [Reading Variants](docs/reading-variants.md) for Parent Name relationships,
136 explicit/resulting Scope, merge preflight and the four original contracts.
137 [Reading Scope](docs/reading-scope.md) for Scope APIs, actual Topic
138 references, merge handles and the three original comparisons.
139 [Reading Typed Topic Names](docs/reading-typed-topic-names.md) covers Slice 12A.
140 [Reading DateTime Timezone Canonicalization](docs/reading-datetime-timezones.md)
141 for UTC conversion, calendar limits, actual stored strings and executable inspection.
142 [Reading DateTime Occurrences](docs/reading-datetime-occurrences.md)
143 for lexical/value/canonical distinctions and the two normative discrepancies.
144 [Reading Date Occurrences](docs/reading-date-occurrences.md)
145 for the ten original Date cases, calendar rules, stored values and inspection.
146 [Reading Decimal Occurrences](docs/reading-decimal-occurrences.md)
147 for the seven Decimal cases, string representation, equality and inspection.
148 [Reading Integer Occurrences](docs/reading-integer-occurrences.md)
149 for integer representation, lexical equality, canonicalization and inspection.
150 [Reading Prefixes and Explicit Datatypes](docs/reading-prefixes-and-datatypes.md)
151 for the seven original cases, current API, object properties and diagnostics.
152 [Reading the First Four CTM Occurrences](docs/reading-first-occurrences.md)
153 for actual parent/type references, literal values, datatypes and executable inspection.
154 [Reading Canonical Topic Maps](docs/reading-canonical-topic-maps.md)
155 for the CXTM API, normative clauses, base-locator policy and actual byte comparisons.
156 [Reading the First CTM Associations](docs/reading-first-ctm-associations.md)
157 for actual Association/Role items, late Topic merges and current subset boundaries.
158 [Reading Topic Identity and Merging](docs/reading-topic-merging.md)
159 for the current API, reference replacement, identifier tails and exact subset
160 boundaries. See [Reading the First Five CTM Documents](docs/reading-first-five-ctm-documents.md)
161 for original sources, actual objects, normative clauses, baseline evidence and
162 copy-pasteable DESCRIBE forms. [API and boundaries](docs/slice-1-api-and-boundaries.md)
163 document the accepted Slice 1 baseline; the Slice 2 document supersedes its
164 statements about absent identity tails, merging and immutable Name references.
165 The Slice 3 document supersedes historical statements about absent Associations/Roles.
166
167 The pinned suite targets the **CTM draft dated 2010-03-31**. Passing it alone
168 does not establish full ISO/IEC 13250-6:2010 conformity.
169
170 ## Load and inspect
171
172 In a new Common Lisp process, with this checkout selected:
173
174 ```lisp
175 (require :asdf)
176 (asdf:initialize-source-registry
177 '(:source-registry :ignore-inherited-configuration))
178 (asdf:load-asd #p"/Users/rgb/workspace/topic-maps/topic-maps.asd")
179 (asdf:load-system "topic-maps/ctm")
180 (asdf:load-system "topic-maps")
181 (defparameter *corpus* (topic-maps.test-foundation:discover-corpus))
182 (topic-maps.test-foundation:summarize-corpus *corpus*)
183 (topic-maps.test-foundation:select-cases *corpus* :category :invalid)
184 (topic-maps.test-foundation:select-cases *corpus* :id "ctm/in/simple.ctm")
185 (asdf:test-system "topic-maps/tests")
186 (asdf:test-system "topic-maps/ctm/tests")
187 (asdf:test-system "topic-maps/ctm/merging-tests")
188 (asdf:test-system "topic-maps/ctm/association-tests")
189 (asdf:test-system "topic-maps/ctm/decimal-tests") ; includes retained Integer/Prefix/Occurrence/CXTM suites
190 (topic-maps.test-foundation:write-validation-report
191 *corpus* #p"/tmp/topic-maps-inventory.json")
192 ```
193
194 Change only the checkout pathname when using another machine. The case's
195 Document IRI remains independent of that pathname. Records are inspectable
196 Common Lisp structures, with actual source/baseline pathnames and explicit
197 contracts, results, statuses, diagnostics and reviewed clauses.
198
199 ## Offline verification
200
201 ```sh
202 # SBCL must already be installed. No network is used by this command.
203 TOPIC_MAPS_REPORT=/tmp/topic-maps-validation.json ./scripts/test.sh
204 ```
205
206 Set `SBCL=/absolute/path/to/sbcl` if needed. The launcher starts a new process,
207 ignores system/user initialization, clears inherited ASDF source configuration
208 and writes FASLs outside the repository. It runs byte-integrity fixtures,
209 Common Lisp infrastructure checks and an independent JSON-schema/content check
210 when a report pathname is supplied. `nix develop path:.` supplies a minimal,
211 pinned SBCL/Python/Git environment; initial environment provisioning may need
212 network access, while test execution after provisioning does not.
213
214 The complete upstream corpus is already vendored, unchanged. If absent in a
215 source distribution, acquire it once:
216
217 ```sh
218 python3 scripts/corpus.py acquire
219 python3 scripts/corpus.py verify
220 ```
221
222 Acquisition uses the exact commit and archive digest in
223 [corpus-lock.json](references/corpus-lock.json), refuses to merge over existing
224 contents and verifies every file. Verification is offline. Valid CTM inputs
225 are selected only under `ctm/in`, invalid ones only under `ctm/invalid`, and
226 CXTM baselines only under `ctm/baseline`. Auxiliary files are reported separately.
227
228 ## Current validation and historical foundation
229
230 The complete pinned corpus remains unchanged: 231 valid CTM inputs, 42 invalid
231 inputs, 231 CXTM baselines and 22 CTM auxiliary files. One hundred sixty-five selected positive
232 cases pass actual CXTM byte comparisons. Twenty-two specific rejection contracts pass.
233 Two executed DateTime contracts remain FAIL because of documented normative
234 baseline discrepancies; 84 contracts remain NOT-RUN. A green regression run
235 preserves those two FAILs and never changes an upstream baseline. Unsupported
236 constructs and I/O errors cannot count as the required CTM rejection.
237
238 The full launcher runs the retained 54 foundation checks, two Python integrity
239 fixtures, 123 retained identity checks, 646 merge checks, 657 Association checks,
240 imported-report validation and selected baseline
241 property evidence, 445 retained CXTM checks, 1,101 Occurrence checks, 150 prefix/datatype checks, 665 integer checks, 715 Decimal checks, 1,806 Date checks, 217 DateTime checks, 1,944 timezone checks, 106 typed-name checks, 293 Scope checks, 324 Variant checks and 238 Topic Reference checks, plus
242 244 general Item Identifier checks, 236 Reification checks, 337 n-ary Association
243 checks, 1,016 Core Subject/type-relationship checks, 297 Template checks and 16,294 independent Unicode NFC oracle
244 comparisons. The oracle needs Python UCD 15.0 or 15.1 (tables use UCD 15.1.0).
245 A report path also produces `<path>.semantics.json` with
246 separate model observations, plus `<path>.merging.json`, `<path>.associations.json`
247 and `<path>.cxtm.json` (complete actual/expected XML and byte counts),
248 plus `<path>.occurrences.json` with separate actual Occurrence objects/properties
249 and `<path>.prefixes.json` with all seven selected observations, plus `<path>.integers.json` for integer objects
250 and independently checked arithmetic probes, plus `<path>.decimals.json` with
251 seven Decimal object observations and 157 independent Decimal arithmetic probes,
252 plus `<path>.dates.json` with five original Date results, five original rejections,
253 396 independent date/calendar probes and 15 century decisions. `<path>.datetimes.json` records five actual DateTime
254 models, 45 independent lexical/canonical probes and two executed baseline
255 discrepancies. `<path>.timezones.json` adds exact UTC, fraction and model-purity
256 probes, including every one of the 1681 permitted signed offset minutes.
257 The merge checks include 128 graph/order variants with an independent reachability
258 oracle. Earlier slices recorded version/relocation comparisons; Slice 17A uses
259 the existing single-runtime environment. `<path>.typed-names.json` records Name value/Parent/Type relations;
260 `<path>.scopes.json` records Scope object relations and an ordering witness.
261 `<path>.variants.json` records Variant/Parent/Scope observations and XML ordering.
262 `<path>.references.json` records the new source/counter/model observations.
263 `<path>.identifiers.json` and `<path>.reification.json` retain their model evidence;
264 `<path>.arity.json` adds actual Role references and twelve canonical XML witnesses.
265 `<path>.relationships.json` records the thirteen original reference graphs and
266 atomic Core Subject conflict observations.
267 `<path>.templates.json` records the 57 model observations and ten specific
268 rejections, including original positions and invocation contexts.
269 See [Slice 19B validation](docs/slice-19b-validation.md) for current evidence.
270 [Slice 12A validation](docs/slice-12a-validation.md) records explicit Name types.
271 [Slice 11 validation](docs/slice-11-validation.md) preserves the prior timezone evidence.
272 [Slice 10 validation](docs/slice-10-validation.md) preserves the earlier DateTime baseline.
273 [Slice 9 validation](docs/slice-9-validation.md) records the accepted Date subset.
274 [Slice 8 validation](docs/slice-8-validation.md) records the accepted Decimal subset.
275 [Slice 7 validation](docs/slice-7-validation.md) records the accepted Integer subset.
276 [Slice 6 validation](docs/slice-6-validation.md) records the preceding accepted subset.
277 [Slice 5 validation](docs/slice-5-validation.md) records the preceding accepted subset.
278 [Slice 4 validation](docs/slice-4-validation.md) records the accepted baseline.
279 [Slice 3 validation](docs/slice-3-validation.md),
280 [imported contracts](reports/slice-1/validation.json) and
281 [semantic observations](reports/slice-1/semantics.json).
282
283 [Slice 0 inventory](docs/inventory.md), [architecture](docs/architecture.md),
284 [coverage](docs/coverage.md) and [validation](docs/validation.md) remain historical
285 foundation records. Their original report/logs are preserved. Their statements
286 about absent reader/model systems describe Slice 0, not the current branch.
287 Slice 1 is built from `9c83db34eb1ba8091b004433f088056717dbaa7e` on
288 `codex/ctm-slice-1`. Slice 2 starts from the accepted Slice 1a commit
289 `fb60afd548d679d83fd8ec76bc5ff384c1e23e5f` on `codex/ctm-slice-2`.
290 Slice 13 starts from `788899aea1a20b9cd8237c6106e6613e180d7362` on
291 `codex/ctm-slice-13`.
292 Slice 14 starts from `aeb017e4f4846f2e6a701e581d20eef84a8b83d9` on
293 `codex/ctm-slice-14`.
294 Slice 11 starts from `8741d155d948c7317945eee423a977940cad995e` on
295 `codex/ctm-slice-11`.
296 Slice 10 starts from `2ad41cd1a945a4ce97824464de40646bb899259e` on
297 `codex/ctm-slice-10`.
298 Slice 9 starts from `195d5f5444d776096b765d04c815c23c3db5f84c` on
299 `codex/ctm-slice-9`.
300 Slice 8 starts from `17dd58775e25ab495e3c83deb4b5a36f5e5a124b` on
301 `codex/ctm-slice-8`.
302 Slice 7 starts from `3924d1eac5a070fe5a6a26683e23ac338793f544` on
303 `codex/ctm-slice-7`.
304 Slice 6 starts from `437ad53ea7539d36db3fa871d0931650b7a6a03e` on
305 `codex/ctm-slice-6`.
306 Slice 5 starts from `8036a2b1ce009218f33d300edcc5f715ec1d1124` on
307 `codex/ctm-slice-5`.
308 Slice 4 starts from `35648c4dcc71e8dc6448b64abcacb438e53d64a2` on
309 `codex/ctm-slice-4`.
310 Slice 3 starts from accepted `66664dbff6275d60dab1bc6a232fa70f7b916b8c` on
311 `codex/ctm-slice-3`. Await review before continuing to the next slice.
312
313 Slice 15 starts from `4f7b4d26d40f8f5566594aa2e3ad6296d44fc4c3` on `codex/ctm-slice-15`.