Runtime contract

• word() and contains() query dictionary-derived lexemes.

• word() reports normalized lexical membership and the lowercase, titlecase, and uppercase forms attested by the artifact. uppercase_only is a convenience property for a known uppercase-only lexeme.

• segment() uses authoritative full coverage, case flags, dynamic programming, and optional corpus rank. It applies surface-case acceptance, so a case-folded word may be known to word() while its observed lowercase segment remains unknown.

• entries() requires the dictionary capability.

• context_domains() and supports_domain() require semantic and full coverage. Their target is a character span: overlapping lexical tokens are excluded, while a target containing no lexical token acts as a virtual boundary and keeps adjacent words eligible at distance 1.

Locale contract

language is the physical lexical language. An optional locale is runtime regional pronunciation and presentation state over the selected artifact. Full registered locale tags such as en-US, en-GB, pt-BR, and pt-PT can supply the base language in CLI commands, while short and underscore aliases remain accepted for compatibility. Locale selection does not change the artifact identity or automatically switch source variants.

Semantic context is soft evidence. Positive evidence is not semantic certainty, and missing evidence is not semantic negation. Capability, coverage, schema, language, and missing-artifact failures have controlled public exceptions.

Requirements Overview

Quality Goals

Stakeholders

Schema 10 freeze and bump policy

Schema 10 is frozen at the v0.4.0 release boundary. The structured contract in lexhint.schema_contract is the reviewable definition of the published SQLite layout. It covers required tables, ordered columns, primary and foreign keys, required indexes, WITHOUT ROWID tables, capability relationships, and persisted format versions. Runtime and managed dataset validation check that contract before capability-specific queries begin. SQLite application_id and user_version identify a Lexhint schema-10 file for diagnostics, while metadata schema_version remains authoritative.

A future change requires a schema bump unless it is proven compatible with existing schema-10 readers. Bump for required table or column additions, removals, or renames; primary-key or runtime-required foreign-key changes; required-index changes that alter query assumptions; incompatible JSON payload encodings; deterministic sense-ID type or anchor changes; or search-index construction changes that an existing reader would interpret differently. Schema 10 also covers the lh1 public sense-ID interpretation, source provenance encoding, and the persisted forms, pronunciations, glosses, topics, tags, examples, synonyms, antonyms, relation tags, and semantic source-topic JSON arrays.

Optional metadata keys, diagnostics, error messages, rendering, additive runtime helpers, compatible performance work, and additional validation do not require a schema bump when stored and queried semantics remain unchanged. A schema bump requires rebuilding and republishing every managed variant. Schema 9 artifacts are never migrated in place.

Business context

Lexical and semantic evidence ──> lexhint ──> consumer interpretation and speech policy
Corpus frequency enrichment ────> lexhint ──> lexical ranking and evidence

The consumer decides what an unknown run, version, or candidate should mean. Lexhint ends at evidence and does not own tokenization, pronunciation, or interpretation precedence.

Technical context

• Wiktextract/Kaikki JSONL supplies lexical and semantic data during builds.

• FrequencyWords enriches existing lexemes with corpus fields.

• A local SQLite artifact is the runtime boundary.

• No service endpoint or daemon is required.

Business Context

Technical Context

Strategy Items

Consumer interface

from lexhint import Lexicon, SemanticDomain

lexicon = Lexicon.from_path("en.sqlite3")
completions = lexicon.complete("comp")
suggestions = lexicon.suggest("complier")
headwords = lexicon.match_headwords("comp*", syntax="glob")
relations = lexicon.relations("colour")
targets = lexicon.resolve_headword("colours")
hits = lexicon.search_definitions("computer program", fields=("glosses",), match="all")

The consumer decides what an unknown run, version, or candidate should mean. Lexhint ends at evidence. Relation following is always explicit and does not alter entries() exact lookup.

Lexical lookup and segmentation

1. The consumer constructs Lexicon from one local SQLite artifact, resolved from the vendored, configured cache, or schema-aware managed dataset path when no override is supplied.

2. Construction validates exact schema version, base language, coverage, and explicit capabilities before queries.

3. An optional locale such as en-US, en-GB, pt-BR, or pt-PT is runtime presentation state. It does not change artifact resolution, physical language, or source variant. Regional source tags are defined once in languages.py and used by runtime ordering and filtering.

4. word() and contains() query lexical keys. complete() performs bounded normalized prefix completion through exact lookup and indexed lexical range queries; it is not fuzzy correction. suggest() uses bounded n-gram candidates, match_headwords() uses safe glob/regex scans, and search_definitions() joins the indexed sense-term table without exposing SQLite details. segment() evaluates known spans using authoritative full coverage, case flags, dynamic programming, and optional corpus rank, while retaining strict surface-case acceptance.

5. Runtime reads do not acquire missing data or write to the artifact.

Dictionary relations

1. Build-time defensive extraction projects only the narrow Lexhint-owned Wiktextract fields. It retains redirects, alt_of, and form_of as typed relation candidates and drops unrelated upstream fields.

2. Schema 10 stores deterministic sense IDs, compact source provenance, and normalized redirect, alternative, form_of, synonym, antonym, hypernym, hyponym, and related headword rows only with the dictionary capability. Sense-level synonyms and antonyms remain attached to their exact senses, while incoming relations use the target index.

3. relations(word) performs bounded exact source lookup. resolve_headword(word) follows only the requested relation types. Neither operation is implicit in entries() or fuzzy search.

Semantic evidence

1. lexhint.semantics maps supported raw source topics to stable SemanticDomain values at build time.

2. Context distances are measured from the target character span. Every lexical token overlapping a non-empty target is excluded. If no lexical token overlaps, the target is a virtual insertion boundary and no real token is discarded.

3. Nearby words are queried in batches. Domain weights receive configurable distance decay, with adjacent eligible tokens at distance 1.

4. Results preserve cue text, character spans, token distance, and contribution weight. The candidate cannot validate itself. Domain results are hints rather than sense-disambiguated semantic certainty, and missing evidence is not negative evidence.

Locale matching is based only on source pronunciation and presentation tags retained in the selected artifact. --locale expresses a registered language/region preference and may fall back to untagged evidence; --region selects one exact normalized source tag. If a locale has only multiple untagged fallback pronunciations, the CLI reports that regional evidence was unavailable. A locale request never fabricates evidence or silently selects another Wiktionary source variant.

The public dictionary API distinguishes sense-scoped relations from unsense-disambiguated headword relations and exposes sense_by_id() and incoming_relations().

Capability-specific schema

Artifact metadata is explicit and self-describing. It identifies the physical language, schema, capabilities, profile, and source provenance, while the managed dataset identity is (language, source_variant, capability_variant, schema_version, dataset_version). Locale is optional runtime presentation and filtering state. It never changes artifact identity or creates en-GB or en-US artifacts. Strict schema equality, not a compatibility range, controls SQLite access.

Schema 10 metadata is explicit and self-describing. lexemes is always present for lexical capability and stores lowercase, titlecase, and uppercase attestation flags exposed by WordEvidence. lexeme_domains exists only for semantic; rich entries, senses, sense_topics, and headword_relations exist only for dictionary; lexeme_ngrams exists for search; and sense_search_terms exists for dictionary plus search. Search and relation metadata record index and row counts, and projections remove claims for excluded structures. Schema 9 artifacts are rejected and must be rebuilt; schema 9 and schema 10 dataset families remain side by side on disk.

Managed dataset variants select capability subsets in a strict chain: lexical, runtime (lexical,semantic), dictionary (lexical,semantic,dictionary), and rich (lexical,semantic,dictionary,search). A dictionary projection supports full entry/sense/topic inspection, explicit relation lookup, rendering, semantic context, and completion while intentionally omitting fuzzy suggestion and indexed definition/reverse search. Only rich includes both search structures.

Source contract and diagnostics

The build consumes a narrow Lexhint-owned TypedDict contract for the fields it intentionally retains: lexical identity, POS, senses, topics, forms, IPA sounds, etymology, examples, synonyms, antonyms, redirects, alt_of, and form_of. Translations, descendants, broader linkage taxonomies, audio URLs, IDs, raw glosses, categories, templates, and unknown fields are not persisted. Extraction diagnostics and the local inspection tool report retained and dropped fields without weakening defensive runtime checks.

Provenance and data lifecycle

Metadata records dictionary_source, dictionary_source_sha256, dictionary_source_format, dictionary_source_contract, frequency_source, and frequency_source_sha256, alongside profile, capabilities, creation time, and builder version. Remote dictionary input is hashed while streamed. Automatic FrequencyWords sources are cached by pinned revision and language, validated against an atomic SHA-256 sidecar, and downloaded through temporary files followed by atomic rename.

Relation decision evidence

The schema 10 benchmark compares a pre-schema-10 relation layout with compact compound-key tables and immutable index finalization. On the smoke profile, the candidate measured 204,800 raw bytes and 36,558 gzip bytes versus 425,984 raw bytes and 96,183 gzip bytes for the baseline. Suggestion and definition-search timings were slower in this two-iteration run, so the measurements are comparative evidence rather than English-dataset estimates.

Errors and offline behavior

Capability, coverage, schema, language, and missing-artifact failures have controlled public exceptions. Offline mode rejects every HTTP(S) build source and permits only local or already validated cached inputs. Frequency acquisition fails the build unless the caller explicitly selects --no-frequency or a custom source. Missing semantic evidence is not semantic negation.

Verification and licensing

Tests cover read-only behavior, no-network guards, segmentation, case attestation, virtual-boundary semantic target anchoring, schema and capability validation, frequency policy, semantic target exclusion, CLI contracts, source extraction, relation extraction/API/CLI/projection, and the managed source-aware capability resolver chain. External dictionary and corpus data remain subject to the obligations documented in DATA_SOURCES.md.

Raw bulk Wiktextract input does not contain Kaikki postprocessed website sense.id values. Lexhint therefore ignores that field, retains sparse senseid and Wikidata provenance when available, and generates a versioned deterministic lh1-<language>-<encoded> sense ID. High-cardinality translations and derived graphs remain optional data rather than core tables.

Explicit immutable managed dataset artifacts

Lexhint treats published datasets as explicit, immutable local artifacts rather than package-installed Python models. The dataset manager stores artifacts by normalized base language, source variant, capability variant, exact schema family, and exact release version under the persistent data directory. Their physical identity is (language, source_variant, capability_variant, schema_version, dataset_version). Downloads stream gzip data, verify manifest hashes, sizes, schema, language, coverage, and capabilities, then atomically install the database and sidecar metadata. Runtime Lexicon construction resolves only installed files and never contacts the network automatically. Resolution first applies explicit language, source, capability, and version filters, then prefers English with native fallback when source is unspecified, then selects the maximal capability artifact and newest equivalent release. Incomparable capability maxima are reported as ambiguous.

Managed variants are capability presets: runtime is the recommended lexical,semantic artifact, dictionary is the lexical,semantic,dictionary projection for dictionary inspection without search indexes, and rich adds search for fuzzy suggestions and indexed definition/reverse search. The strict capability chain keeps automatic selection unambiguous.

Quality Requirements Overview

Quality Scenarios

Risk Overview