Architecture Documentation

Generated from archledger records. Do not edit this generated file directly.

Introduction and Goals

Lexhint is a small Python runtime plus prebuilt SQLite evidence artifacts. It supplies lexical membership, optional corpus commonness, compact-string segmentation, stable semantic-domain evidence, and optional rich dictionary inspection.

It does not decide how text is spoken. Spokenform and other consumers own tokenization policy, URLs, numbers, versions, acronyms, pronunciation, and interpretation precedence. Dataset publication is outside this repository.

Runtime contract

word() and contains() query dictionary-derived lexemes.
word() reports normalized lexical membership and the lowercase, titlecase, and uppercase forms attested by the artifact. uppercase_only is a convenience property for a known uppercase-only lexeme.
segment() uses authoritative full coverage, case flags, dynamic programming, and optional corpus rank. It applies surface-case acceptance, so a case-folded word may be known to word() while its observed lowercase segment remains unknown.
entries() requires the dictionary capability.
context_domains() and supports_domain() require semantic and full coverage. Their target is a character span: overlapping lexical tokens are excluded, while a target containing no lexical token acts as a virtual boundary and keeps adjacent words eligible at distance 1.

Semantic context is soft evidence. Positive evidence is not semantic certainty, and missing evidence is not semantic negation. Capability, coverage, schema, language, and missing-artifact failures have controlled public exceptions.

Requirements Overview

Quality Goals

Stakeholders

Architecture Constraints

The architecture is constrained by a local, self-describing SQLite artifact and by the external sources used to build it.

lexhint.Lexicon opens artifacts through SQLite read-only mode.
Runtime operations never fetch network resources, create missing lexemes, or write partial caches.
The CLI resolves default cached or vendored artifacts for ordinary reads and exposes dictionary status for current SQL counts.
SCHEMA_VERSION is an exact artifact compatibility key. Current schema 7 clients select and open only schema 7 artifacts; schema families are stored side by side under s<schema> paths.
Metadata records schema version, base language, coverage, profile, capabilities, creation time, builder version, and source provenance.
lexemes is present for the lexical capability. Semantic and dictionary tables are capability-specific.
Default builds select lexical,semantic,dictionary and automatic pinned full FrequencyWords enrichment.
Frequency is enrichment, not a capability.
External dictionary and corpus data remain separate from the Apache-2.0 code and retain their licensing obligations.

Context and Scope

Lexhint sits between lexical data artifacts and a text-normalization or speech consumer.

Business context

Lexical and semantic evidence ──> lexhint ──> consumer interpretation and speech policy
Corpus frequency enrichment ────> lexhint ──> lexical ranking and evidence

The consumer decides what an unknown run, version, or candidate should mean. Lexhint ends at evidence and does not own tokenization, pronunciation, or interpretation precedence.

Technical context

Wiktextract/Kaikki JSONL supplies lexical and semantic data during builds.
FrequencyWords enriches existing lexemes with corpus fields.
A local SQLite artifact is the runtime boundary.
No service endpoint or daemon is required.

Business Context

Technical Context

Solution Strategy

The solution is organized around a small, explicit evidence pipeline.

Resolve canonical capabilities, profile, frequency mode, source paths, and offline or refresh policy in an immutable build plan before schema creation.
Keep lexical membership and corpus frequency evidence independent from semantic dictionary evidence.
Build rich dictionary tables only for the dictionary capability and materialize topic projections only for semantic.
Use authoritative full coverage, case flags, and dynamic programming for compact-string segmentation.
Exclude every token overlapping the target span from semantic context scoring.
Query nearby context words in batches and apply bounded distance decay to explicit domain evidence.
Validate pinned source hashes and use temporary files followed by atomic rename for downloaded and rebuilt artifacts.
Keep the consumer boundary narrow. Speech pronunciation rules remain downstream.
Emit build configuration and progress on stderr so successful JSON output remains a single stdout document.

Strategy Items

Building Block View

The package is organized around a local artifact runtime and focused build modules.

lexhint.lexicon.Lexicon owns read-only artifact access, lexical lookup, prefix completion, segmentation, dictionary inspection, and semantic evidence queries.
lexhint.schema defines schema and capability validation for the self-describing SQLite artifact.
lexhint.builder creates fresh atomic artifacts from streamed source data and applies the immutable build plan.
lexhint.extract converts source records into curated lexical and dictionary data.
lexhint.semantics projects source topics into the stable SemanticDomain taxonomy.
lexhint.frequency and lexhint.sources resolve corpus enrichment and source provenance.
lexhint.store persists lexemes, domains, rich dictionary tables, metadata, and indexes.
lexhint.cli exposes build and runtime operations in human-readable and JSON forms.

The public package exports Lexicon and SemanticDomain as the principal consumer interface. It also exports SCHEMA_VERSION, DATASET_VARIANTS, DATASET_VARIANT_NAMES, DEFAULT_DATASET_VARIANT, and supported_base_languages() for the separate dataset publisher contract. Build and source helpers remain available from their owning modules.

Consumer interface

from lexhint import Lexicon, SemanticDomain

lexicon = Lexicon.from_path("en.sqlite3")
completions = lexicon.complete("comp")
segments = lexicon.segment("chatgpt")
text = "The compiler is 8.3.2."
start = text.index("8.3.2")
evidence = lexicon.supports_domain(
    text, target=(start, start + len("8.3.2")), domain=SemanticDomain.COMPUTING
)

The consumer decides what an unknown run, version, or candidate should mean. Lexhint ends at evidence.

Runtime View

Lexical lookup and segmentation

The consumer constructs Lexicon from one local SQLite artifact, resolved from the vendored, configured cache, or schema-aware managed dataset path when no override is supplied.
Construction validates exact schema version, base language, coverage, and explicit capabilities before queries.
An optional locale such as GB or US is runtime presentation state. It does not change artifact resolution or physical English dataset identity.
word() and contains() query dictionary-derived lexemes. word() exposes casing attestation already stored in the lexeme row. complete() performs bounded normalized prefix completion through exact lookup and indexed lexical range queries. segment() evaluates known spans using authoritative full coverage, case flags, dynamic programming, and optional corpus rank, while retaining strict surface-case acceptance.
Runtime reads do not acquire missing data or write to the artifact.

Semantic evidence

lexhint.semantics maps supported raw source topics to stable SemanticDomain values at build time.
Context distances are measured from the target character span. Every lexical token overlapping a non-empty target is excluded. If no lexical token overlaps, the target is a virtual insertion boundary and no real token is discarded.
Nearby words are queried in batches. Domain weights receive configurable distance decay, with adjacent eligible tokens at distance 1.
Results preserve cue text, character spans, token distance, and contribution weight. The candidate cannot validate itself. Domain results are hints rather than sense-disambiguated semantic certainty, and missing evidence is not negative evidence.

Deployment View

Lexhint is deployed as a local Python package and a local SQLite evidence artifact. There is no application server, worker, or persistent service.

Consumers install the package and open an artifact with Lexicon.from_path() or the default Lexicon resolution.
Artifact paths are selected by the caller or build workflow; CLI --path is an explicit override.
dictionary status reports current row counts and metadata without rebuilding the artifact.
A complete local artifact supports offline lexical, segmentation, dictionary, and semantic reads when the corresponding capabilities are present.
Generated artifacts contain source and hash provenance for dictionary and corpus inputs.
Build downloads and replacements use temporary files and atomic rename.
Generated external datasets are distributed separately from code according to DATA_SOURCES.md.

Cross-cutting Concepts

Capability-specific schema

Schema metadata is explicit and self-describing. language, locale, variant, schema_version, and dataset_version remain separate dimensions. Locale is optional and does not create en-GB or en-US artifacts. Strict equality, not a compatibility range, controls SQLite access.

Schema 7 metadata is explicit and self-describing. lexemes is always present for lexical capability and already stores lowercase, titlecase, and uppercase attestation flags exposed by WordEvidence. lexeme_domains exists only for semantic; each row stores bounded deterministic weight and source-topic provenance. Rich entries, senses, sense_topics, forms, and pronunciations exist only for dictionary. Old partial-cache schemas are rejected and must be rebuilt. Prefix completion uses the existing lexemes.word primary-key range and does not require a schema change.

Provenance and data lifecycle

Metadata records dictionary_source, dictionary_source_sha256, frequency_source, and frequency_source_sha256, alongside profile, capabilities, creation time, and builder version. Remote dictionary input is hashed while streamed. Automatic FrequencyWords sources are cached by pinned revision and language, validated against an atomic SHA-256 sidecar, and downloaded through temporary files followed by atomic rename.

Errors and offline behavior

Capability, coverage, schema, language, and missing-artifact failures have controlled public exceptions. Offline mode rejects every HTTP(S) build source and permits only local or already validated cached inputs. Frequency acquisition fails the build unless the caller explicitly selects --no-frequency or a custom source. Missing semantic evidence is not semantic negation.

Verification and licensing

Tests cover read-only behavior, no-network guards, segmentation, case attestation, virtual-boundary semantic target anchoring, schema and capability validation, frequency policy, semantic target exclusion, CLI contracts, and source extraction. External dictionary and corpus data remain subject to the obligations documented in DATA_SOURCES.md.

Explicit immutable managed dataset artifacts

Lexhint treats published datasets as explicit, immutable local artifacts rather than package-installed Python models. The dataset manager stores artifacts by normalized base language, capability variant, exact schema family, and exact release version under the persistent data directory. Downloads stream gzip data, verify manifest hashes, sizes, schema, language, coverage, and capabilities, then atomically install the database and sidecar metadata. Runtime Lexicon construction resolves only installed files and never contacts the network automatically. The highest-capability compatible installed variant is selected by default, while callers may pin a variant and release version.

Architecture Decisions

The current architecture records these decisions.

Use a self-describing SQLite artifact. Schema, language, coverage, profile, capabilities, and provenance are validated at runtime.
Separate lexical, semantic, and dictionary capabilities. Consumers can select the evidence they need without allowing data from an older artifact to leak into a fresh build.
Use indexed lexical ranges for completion. complete() is a local read-only normalized prefix query with exact-match priority and explicit frequency or lexical ordering.
Treat frequency as enrichment. Corpus rank improves segmentation and commonness evidence but does not define lexical capability.
Build fresh artifacts atomically. Capability-specific tables are created from the resolved build plan and replacements cannot expose partial output.
Use stable semantic domains. Raw source topics are projected into a small deterministic taxonomy at build time.
Expose case attestation without weakening segmentation. word() exposes normalized membership and stored case forms, while segment() retains surface-case acceptance so consumers can apply context-specific policy.
Anchor semantic context to character spans. Overlapping target tokens are excluded; a target with no lexical token is a virtual boundary whose adjacent cues remain eligible at distance 1.
Treat semantic context as soft evidence. Lexhint reports explainable hints, not sense disambiguation or semantic certainty.
Keep the runtime read-only and offline by default. Acquisition belongs to explicit build workflows.
Keep a narrow consumer boundary. Lexhint supplies evidence; downstream consumers own interpretation and speech rendering.

Quality Requirements

Quality attribute	Architectural response	Observable scenario
Correctness	Explicit schema and capability validation, case-aware lexemes, authoritative full coverage, and target exclusion	A candidate is never used as its own semantic cue and incompatible artifacts fail at construction.
Determinism	Stable source projections, bounded weights, immutable build plans, and explicit result fields	Repeating a local query returns the same segments and evidence ordering.
Performance	Indexed SQLite lookups, bounded context windows, batched nearby-word queries, and streamed builds	A context query evaluates only the bounded local evidence window.
Resilience	Read-only runtime access, source hashes, temporary downloads, and atomic replacement	A failed build does not replace an existing artifact with partial output.
Maintainability	Focused runtime and build modules, capability-specific schema, and boundary tests	Schema, extraction, semantic projection, storage, and CLI behavior can be checked independently.
Compliance	External resources remain separate from code and provenance is embedded in artifacts	A distributor can review data obligations before distributing generated artifacts.

Quality Requirements Overview

Quality Scenarios

Risks and Technical Debt

Upstream availability and format drift. FrequencyWords and Wiktextract/Kaikki remain external inputs. Hash validation and source checks reduce silent corruption, but upstream changes can still prevent builds.
External data licensing. Generated artifacts inherit obligations from their dictionary and corpus sources. Redistribution must follow DATA_SOURCES.md.
Heuristic segmentation. Dynamic-programming segmentation is evidence, not linguistic analysis, and may need tuning for new languages or identifier styles.
Incomplete capability coverage. A consumer cannot use dictionary or semantic operations when the artifact lacks those capabilities. Missing evidence must not be treated as semantic negation.
Schema evolution. Incompatible schema or capability changes require rebuilding artifacts.
Scope boundary. Lexhint does not pronounce text, tokenize all consumer inputs, or resolve interpretation precedence. Those responsibilities remain downstream.

Risk Overview

Glossary

Lexicon: The read-only runtime that opens one self-describing SQLite evidence artifact.
Lexeme: A dictionary-derived lexical row with case flags and optional corpus fields.
SemanticDomain: A stable taxonomy value projected from supported source topics.
Capability: An explicit artifact feature such as lexical, semantic, or dictionary.
Full coverage: An authoritative artifact suitable for segmentation and semantic context queries.
Context cue: A nearby non-target token whose domain evidence contributes a bounded score.
Target span: The source character interval excluded from semantic context evidence.
Wiktextract/Kaikki: The upstream dictionary data used for lexical, semantic, and rich dictionary builds.
FrequencyWords: The upstream corpus source used to enrich existing lexemes with commonness fields.
Spokenform: A downstream consumer that owns tokenization, interpretation, pronunciation, and speech policy.