Lexphon architecture

Boundaries

The runtime ownership chain is:

g2lex-data -> G2Lex -> Lexphon -> application adapters

g2lex-data is the producer and publisher. It obtains, transforms, validates, licenses, reproduces, and publishes immutable generic pronunciation data releases. G2Lex is the storage primitive that represents, packs, opens, and queries typed lexicon data. Lexphon is the generic runtime consumer. KokoroG2P is a downstream model adapter.

Lexphon owns catalog consumption, explicit verified installation, immutable local asset storage, language profiles, candidate generation, ordered layered lookup, selectors, pronunciation alphabet normalization, token provenance, CLI behavior, and optional generic pronunciation providers.

Lexphon owns generic pronunciation providers as well as lexicon pronunciation cleanup. eSpeak and Goruut return raw source pronunciation; Lexphon normalizes it once into clean IPA and structured provenance, including provider, requested provider language, raw source pronunciation, and removed language-control markers. No Kokoro vocabulary conversion or model-specific normalization occurs here.

The eSpeak provider delegates runtime discovery, native/CLI selection, voice resolution, execution, batching, version diagnostics, and cleanup to espeakng-runtime. Lexphon retains provider policy, language normalization, U+200D tied IPA selection, raw-output normalization, typed errors, and provenance.

Phonemizer.lookup_lexicon() is the lexicon-evidence operation and never invokes a provider. Phonemizer.lookup() is the staged pronunciation operation: it first calls lookup_lexicon(), then invokes the configured provider only after a lexicon miss. Provider results use source="provider" and identify the provider separately. Lexphon does not own source acquisition, dataset transformations, licensing transformations, production dictionary build recipes, Kokoro vocabulary, Kokoro stress or rating policy, or hidden downloads. Lexphon must never contain source acquisition or G2Lex build recipes for production dictionaries.

Catalog and installation

Lexphon accepts catalog version 1 with runtime contract g2lex-data.catalog.v1. Every artifact has a validated logical ID, locale, kind, source encoding, data version, release tag, manifest reference, asset reference, hashes, and asset size. Membership assets use none and are not pronunciation layers. Pronunciation assets may declare application-specific encodings such as kokoro-v1; the data layer can discover, install, and verify them even when the generic Phonemizer cannot normalize them.

DataStore.install() is the only asset download path and is an explicit network-capable provisioning operation. It downloads and verifies the manifest, validates manifest metadata against the catalog, downloads and verifies the asset, opens it with G2Lex, and moves a complete version into assets/<logical-id>__<name>/<data-version>/. A logical ID and data version are immutable. The index is updated atomically. Failed installations remove staging and never register an active partial version.

Remote retrieval and downloaded-data integrity are separate failure classes. The public exception taxonomy is:

Catalog parsing/contract errors -> CatalogError
Remote retrieval/publication errors -> DataDownloadError
Downloaded/local contract violations -> DataIntegrityError
Missing local installation -> LexiconNotInstalledError
Installed but unusable pronunciation layer -> LexiconNotUsableError

A catalog can declare an artifact before its referenced release resource is published. available reports that declaration without reachability checks. install reports retrieval context such as resource, release tag, data version, URL, and HTTP status through DataDownloadError; hash, metadata, and G2Lex validation failures remain DataIntegrityError. installed.json stores the local manifest and asset paths plus language, kind, encoding, versions, release tag, hashes, size, and logical hash. path(), metadata(), verify(), and remove() use only this local state. A copied store can therefore be opened offline. Phonemizer, lookup, token phonemization, and rendering do not load catalogs or access the network.

Data release versions and the Lexphon Python package version are independent. Applications should pin both the Python dependency and the immutable data catalog or release used during provisioning.

LexHint source variants

*:lexhint is the preferred/default logical asset. For non-English languages with both upstream sources, it is English-Wiktionary-derived, while *:lexhint-native is the explicit native-Wiktionary alternative. Native-only languages such as Thai use only *:lexhint-native. English is the special case: en-us:lexhint and en-gb:lexhint use native English Wiktionary.

Logical IDs already encode source selection, so Lexphon does not resolve source variants or add a source selector. kokoro-v1 artifacts may be discoverable, installable, and verifiable through DataStore, but they are not generic Phonemizer layers. Kokoro-specific conversion remains a KokoroG2P responsibility.

Runtime rule

Phonemizer(...) opens only already-installed pronunciation assets. A missing lexicon raises LexiconNotInstalledError with an install command. Construction, lookup, token phonemization, and rendering do not download data.

Selected layers are searched in caller order. Within each layer, profile-ordered candidates are searched before moving to the next layer. Profiles provide generic Unicode normalization, apostrophe normalization, and exact, lower, casefold, or title candidates. The built-in generic defaults are en-us:lexhint, en-gb:lexhint, fr:lexhint, and th:lexhint-native for the affected profiles; German remains de-de:gold, and CMUdict remains an explicit en-us:cmudict alternative. A language-only asset such as fr:lexhint may serve fr-FR, but a region-specific asset cannot cross to another region.

The public result alphabet is IPA and every non-null pronunciation is Unicode NFC. ARPABET is converted with deterministic phone and primary or secondary stress rules. Unknown phones and unsupported encodings raise UnsupportedAlphabetError. Lexphon also consumes recognized inline language-control markers in IPA source notation at this boundary. The marker text is removed from public IPA and preserved as PronunciationLanguageMarker metadata with offsets into the cleaned IPA.

Structured tokens preserve the original token, selected pronunciation, source category, logical lexicon, matched key, source encoding, structured variants, selector context, provider and requested language, punctuation state, and pronunciation provenance. PronunciationToken.pronunciation, source_pronunciation, and language_markers derive from the first variant. Unknown words remain explicit when no provider returns a pronunciation.

KokoroG2P integration

KokoroG2P should import the Lexphon Python API, not spawn the CLI:

from lexphon import Phonemizer

engine = Phonemizer(
    "de-DE",
    lexicons=["de-de:gold"],
    fallback="espeak",
)

lexical = engine.lookup_lexicon("File")
actual = engine.lookup("File")

lexical is routing evidence. actual is the pronunciation result. A provider result has source="provider", while a lexical result has source="lexicon".

KokoroG2P converts generic Lexphon IPA to its model vocabulary and applies model-specific ratings, stress, and compatibility policy. Lexphon remains independent of KokoroG2P and does not validate or rewrite the Kokoro inventory.