Span-Based Phonemization Guide

This document explains the span-based (offset-based) phonemization system in kokorog2p, which provides deterministic, pipeline-friendly text-to-phoneme conversion.

Overview

The span-based system uses character offsets to precisely identify and override specific portions of text during phonemization. This approach is more robust than word-based matching because it handles:

Duplicate words with different pronunciations (e.g., “the cat the dog”)
Partial word matches (e.g., distinguishing “cat” from “category”)
Complex text structures with punctuation and whitespace variations
Pipeline integration with offset-preserving preprocessing

Core Types

TokenSpan

Represents a single token with its phonemization and metadata:

@dataclass
class TokenSpan:
    text: str              # Original text ("hello")
    char_start: int        # Character offset start (0)
    char_end: int          # Character offset end (5)
    lang: str | None = None  # Language override ("en-us", "fr", etc.)
    extended_text: str | None = None  # Expanded text for phonemization
    meta: dict[str, Any] = field(default_factory=dict)  # Metadata including phonemes

Key Properties:

char_start and char_end are character offsets in the clean text (after markup removal)
char_end is exclusive (Python slice convention: text[char_start:char_end])
Multiple tokens can reference the same text position if tokenization creates sub-parts
Whitespace is inferred from offsets (tokens are words/punctuation only)
Abbreviations keep trailing periods in the same token (e.g., Mr.)
extended_text holds optional expansions (e.g., Mr. → Mister or 1 → one)
Phonemes are stored in meta["phonemes"] after phonemization

Example:

text = "Hello Mr. Smith!"
tokens = tokenize(text, language="en-us")
# TokenSpan(text="Hello", char_start=0, char_end=5, ...)
# TokenSpan(text="Mr.", char_start=6, char_end=9, ...)
# TokenSpan(text="Smith", char_start=10, char_end=15, ...)
# TokenSpan(text="!", char_start=15, char_end=16, ...)

OverrideSpan

Specifies a region of text to override during phonemization:

@dataclass
class OverrideSpan:
    char_start: int                 # Character offset start
    char_end: int                   # Character offset end (exclusive)
    attrs: dict[str, str]           # Override attributes

Common Attributes:

ph: Direct phoneme override (e.g., {"ph": "həlˈO"})
lang: Language switch (e.g., {"lang": "fr"})
Custom attributes for pipeline processing

Example:

text = "the cat the dog"
overrides = [
    OverrideSpan(0, 3, {"ph": "ðə"}),   # First "the" → /ðə/
    OverrideSpan(8, 11, {"ph": "ði"}),  # Second "the" → /ði/
]

Exact sub-token language spans

The default overlap="snap" behavior is unchanged. Opt into exact source-aligned fragments with overlap="split":

from kokorog2p import OverrideSpan, phonemize_prepared

result = phonemize_prepared(
    "Manpowerdiskussion",
    language="de",
    overlap="split",
    overrides=[OverrideSpan(0, 8, {"lang": "en"})],
)

Each fragment satisfies fragment.text == source[fragment.char_start:fragment.char_end]. Explicit ph and lang overrides take precedence over automatic routing.

Automatic language routing

Enable routing with an explicit candidate allowlist:

result = phonemize_prepared(
    text,
    language="de",
    language_routing={"mode": "auto", "languages": ["de", "en"]},
)

Routing uses positive evidence from the effective selected lexical resources, keeps the explicit document language for collisions and ambiguity, and exposes LanguageRoute records in result.language_routes. A hit is not exclusive ownership, so a word present in multiple selected stacks stays in the default language.

Evidence-capable resources are:

Externally provisioned G2Lex: English US, GB English, and French.
Provisioned Lexphon: German, Portuguese BR/PT, Russian, Thai, Vietnamese, Japanese, Korean, and Swedish when NST is explicitly selected.

Spanish, Italian, Czech, Hebrew, Arabic, Chinese, Kazakh, and Hindi still phonemize normally, but they have no selected lexical evidence provider and cannot positively claim a foreign token. Generic pronunciation, rules, eSpeak, Goruut, pypinyin, Phonikud, g2pK, pyopenjtalk, and fallback paths are never used as evidence. Candidate frontends are lazy and may be supplied with g2p_resolver; default foreign frontends do not inherit default-language lexicon or language-specific options. Hindi eSpeak pronunciation is realization data, not lexical evidence for automatic routing.

Use target_model="1.0" to constrain every automatic candidate and the final token IDs to one fixed Kokoro vocabulary. If evidence exists but the routed pronunciation is incompatible, the route is rejected with a diagnostic and the default route is used. Routing changes only G2P frontend selection. It never selects an acoustic model. Explicit ph, phonemes, lang, and language overrides take precedence.

With Lexphon 0.1.3, German evidence also carries structured pronunciation language markers. The DE/EN analyzer uses those markers only for bounded, marker-backed loanword rules. Thus an exact German hit such as starten remains German when neither the whole word nor the candidate stem is marked as English, while marked forms such as gecancelt and downloaden can expose English fragments. Raw Lexphon source notation is producer-owned and is not parsed by KokoroG2P.

Structured stress overrides

Use stress in an OverrideSpan to change the relative stress of the resolved phonemes for a token or span:

from kokorog2p import OverrideSpan, phonemize

result = phonemize(
    "zwei Minuten",
    language="de",
    overrides=[
        OverrideSpan(0, 4, {"stress": "+2"}),
    ],
)

Supported values are strings:

"-2" removes ˌ and ˈ markers.
"-1" lowers primary stress to secondary stress.
"+1" adds secondary stress or promotes existing secondary stress.
"+2" adds or promotes primary stress.

Stress is applied after lang, G2P, lexicon, fallback, and ph resolution. A span covering several lexical tokens applies the same relative stress independently to each spoken token. Punctuation, whitespace, and empty phoneme output are unchanged. A one-word ph override may combine with stress. A collapsed multi-word ph override is diagnosed instead of guessing which word receives stress.

This structured feature changes phoneme stress markers only. It is not direct volume, pitch, duration, SSML emphasis, or general prosody control. Markdown, Misaki, and Kokoro-compatible stress syntax are outside this API.

PhonemizeResult

The complete phonemization output:

@dataclass
class PhonemizeResult:
    clean_text: str                  # Text with markup removed
    tokens: list[TokenSpan]          # Token-level information with offsets
    extended_text: str               # Expanded text for phonemization
    phonemes: str                    # Concatenated phoneme string
    token_ids: list[int]             # token IDs for model input
    warnings: list[str]              # Alignment warnings

Prepared phonemization

phonemize_prepared() is the explicit core entry point. phonemize() remains a prepared-text compatibility spelling; neither entry point performs written-to-spoken semantic expansion:

from kokorog2p import phonemize_prepared

result = phonemize_prepared(
    prepared_text,
    language="de",
    overrides=prepared_coordinate_overrides,
    return_phonemes=True,
    return_ids=True,
 )

The supplied prepared text remains the coordinate space for clean_text, tokens, and offsets. KokoroG2P performs tokenization, explicit language/phoneme overrides, G2P/backend normalization, punctuation handling, and token-ID generation. Semantic preparation is owned by the caller.

Extended Text Layer

extended_text is retained as a result field for compatibility and normally equals the prepared input. The core does not populate it with number, abbreviation, date, unit, or currency expansions.

SSMD and phrasplit compatibility

SSMD AnnotationSpan objects and phrasplit SplitSegment objects use the same coordinate contract as kokorog2p: zero-based, half-open offsets relative to the cleaned text. kokorog2p accepts compatible objects by structure; they do not need to inherit from or implement kokorog2p classes.

Normalize SSMD annotations

Use the public adapter before applying annotations:

from kokorog2p.integrations import overrides_from_ssmd

overrides = overrides_from_ssmd(parsed.annotations)

The normalization rules are:

Input attribute	kokorog2p attribute	Behavior
`ph`	`ph`	Used as a direct phoneme override
`ipa`	`ph`	Mapped to a direct phoneme override
`lang` or `language`	`lang`	Used as a per-span language override
`tag`, `node`, `node_id`, `kind`	omitted	SSMD dispatch metadata is not copied
`sampa` or X-SAMPA alphabet	—	Rejected explicitly; never treated as IPA

The adapter copies attribute dictionaries and validates integer offsets. Pass text_length= when document bounds should also be checked.

Rebase document spans to sentence segments

SSMD annotations are usually relative to the complete ParseSpansResult.clean_text, while phonemize(segment.text) requires segment-local offsets. Intersect and rebase by coordinates:

from kokorog2p import phonemize
from kokorog2p.integrations import overrides_for_segment, overrides_from_ssmd

parsed = ssmd.parse_spans(source)
overrides = overrides_from_ssmd(parsed.annotations)
segments = phrasplit.split_with_offsets(
    parsed.clean_text, mode="sentence", language="en", use_spacy=None
)

results = []
for segment in segments:
    assert parsed.clean_text[segment.char_start:segment.char_end] == segment.text
    results.append(
        phonemize(
            segment.text,
            overrides=overrides_for_segment(
                segment.char_start, segment.char_end, overrides
            ),
            use_spacy=None,
        )
    )

overrides_for_segment() preserves partial intersections and returns new OverrideSpan objects. It handles duplicate sentences, punctuation, and whitespace gaps without searching for segment text. phonemize_segments() provides the same dependency-free orchestration when a caller already has a phonemize callable.

Character Offset Coordinate System

Basic Rules

Zero-indexed: First character is at position 0
Exclusive end: Range [start, end) means characters from start up to but not including end
Clean text reference: Offsets refer to text after markup removal but before normalization

Examples

text = "Hello world"
#       0123456789...

# "Hello" → char_start=0, char_end=5
# "world" → char_start=6, char_end=11
# " " (space) → char_start=5, char_end=6

With Markup

# Original: "@Hello@ world"  (with marker-based override)
# Clean text: "Hello world"
#              0123456789...

override = OverrideSpan(0, 5, {"ph": "həlˈO"})  # Refers to "Hello" in clean text

Duplicate Words

text = "the cat the dog"
#       012345678901234...

# First "the" → OverrideSpan(0, 3, ...)
# Second "the" → OverrideSpan(8, 11, ...)

Alignment Modes

The system supports two alignment modes for applying overrides to tokens:

1. Span Alignment (Default)

Recommended for all new code. Uses character offsets for deterministic matching.

Matching Logic:

Exact match: Override span exactly matches token span → Apply override
Partial overlap (snap mode): Override partially overlaps token → Apply with warning
No overlap: Override doesn’t touch token → Skip

Advantages:

✅ Handles duplicate words correctly
✅ No ambiguity in complex text
✅ Predictable behavior for pipeline integration
✅ Works with partial word matches

Example:

result = phonemize(
    "the cat the dog",
    overrides=[
        OverrideSpan(0, 3, {"ph": "ðə"}),
        OverrideSpan(8, 11, {"ph": "ði"}),
    ],
    alignment="span"  # default
)
# Both overrides applied to correct "the" instances

2. Legacy Word Alignment

Deprecated. Uses word-text matching (first occurrence).

Matching Logic:

Finds first token with matching text
Cannot distinguish between duplicate words
Provided only for backward compatibility

Limitations:

❌ Cannot handle duplicate words with different overrides
❌ Order-dependent behavior
❌ Fragile with whitespace/punctuation variations

Example:

result = phonemize(
    "the cat the dog",
    overrides=[
        OverrideSpan(0, 3, {"ph": "ðə"}),
        OverrideSpan(8, 11, {"ph": "ði"}),  # Will NOT work correctly!
    ],
    alignment="legacy"
)
# Only first "the" gets overridden (both overrides apply to same word)

Overlap Handling

When an override partially overlaps with a token, the system can handle it in two ways via the overlap parameter in phonemize():

Snap Mode (Default)

Apply the override and emit a warning:

result = phonemize(
    "category",
    overrides=[OverrideSpan(0, 3, {"ph": "kæt"})],  # "cat" is only part of "category"
    overlap="snap"
)
# Override applied to entire "category" token
# Warning: "Override span (0, 3) partially overlaps token boundaries; snapping to tokens..."

Strict Mode

Skip the override and emit a warning on partial overlap:

result = phonemize(
    "category",
    overrides=[OverrideSpan(0, 3, {"ph": "kæt"})],
    overlap="strict"
)
# Override skipped
# Warning: "Override span (0, 3) partially overlaps token boundaries; skipping (strict mode)"

Language Switching

Override spans can specify language changes for specific text regions:

# Mix English and French in same sentence
text = "Hello Bonjour world"
overrides = [
    OverrideSpan(6, 13, {"lang": "fr"})  # "Bonjour" in French
]

result = phonemize(text, language="en-us", overrides=overrides)
# "Hello" → English G2P
# "Bonjour" → French G2P
# "world" → English G2P

Language Codes:

Use standard language codes: en-us, fr, de, es, etc.
See get_g2p() for supported languages

Phoneme Overrides

Direct phoneme replacement bypasses G2P processing:

text = "read the book"
overrides = [
    OverrideSpan(0, 4, {"ph": "ɹˈEd"}),  # "read" as past tense
]

result = phonemize(text, language="en-us", overrides=overrides)
# Uses provided phonemes for "read" instead of G2P lookup

Phoneme Override Priority: If both ph and lang are specified, ph takes precedence:

OverrideSpan(0, 5, {"ph": "test", "lang": "fr"})
# "ph" is used, "lang" is ignored for this span

Custom Attributes

Override spans can carry custom attributes for downstream processing:

overrides = [
    OverrideSpan(0, 5, {
        "ph": "həlˈO",
        "speaker": "male",
        "emphasis": "strong"
    })
]

result = phonemize(text, language="en-us", overrides=overrides)
# Custom attributes stored in token.meta
for token in result.tokens:
    print(token.meta)  # {"ph": "həlˈO", "speaker": "male", "emphasis": "strong", ...}

Best Practices

1. Use Span Alignment (Default)

Always use span-based alignment unless you have a specific reason to use legacy mode:

# ✅ Good
result = phonemize(text, language="en-us", overrides=overrides)

# ❌ Avoid (unless backward compatibility required)
result = phonemize(text, language="en-us", overrides=overrides, alignment="legacy")

2. Compute Offsets from Clean Text

Always compute offsets from the text after markup removal:

# Original markup text
markup_text = "[Hello]{ph='test'} world"

# Remove markup to get clean text
clean_text = "Hello world"

# Compute offsets from clean text
override = OverrideSpan(0, 5, {"ph": "həlˈO"})  # Refers to "Hello" in clean_text

3. Use tokenize() for Debugging

Inspect tokenization to understand offset positions:

from kokorog2p import tokenize

text = "the cat the dog"
tokens = tokenize(text, language="en-us")

for tok in tokens:
    print(f"{tok.text!r} → [{tok.char_start}:{tok.char_end}]")
# 'the' → [0:3]
# ' ' → [3:4]
# 'cat' → [4:7]
# ...

4. Check Warnings

Always inspect result.warnings to catch alignment issues:

result = phonemize(text, language="en-us", overrides=overrides)

if result.warnings:
    print("Alignment warnings:")
    for warning in result.warnings:
        print(f"  - {warning}")

5. Test with Duplicates

Always test your override logic with duplicate words:

# ✅ Good test case
text = "the cat saw the dog"
overrides = [
    OverrideSpan(0, 3, {"ph": "ðə"}),    # First "the"
    OverrideSpan(12, 15, {"ph": "ði"}),  # Second "the"
]
result = phonemize(text, language="en-us", overrides=overrides)
assert len(result.warnings) == 0

Common Pitfalls

❌ Using Word Offsets Instead of Character Offsets

# WRONG: These are word positions, not character offsets
override = OverrideSpan(0, 1, {"ph": "test"})  # Trying to select first word

# RIGHT: Use character positions
text = "hello world"
override = OverrideSpan(0, 5, {"ph": "test"})  # "hello" is chars 0-5

❌ Ignoring Clean Text vs Original Text

# Original with markers: "@Hello@ world"
# Clean: "Hello world"

# WRONG: Offsets based on original text with markers
override = OverrideSpan(0, 7, {"ph": "test"})  # Includes marker chars

# RIGHT: Offsets based on clean text
override = OverrideSpan(0, 5, {"ph": "test"})  # Just "Hello"

❌ Forgetting Exclusive End

text = "hello"
#       01234

# WRONG: Inclusive end
override = OverrideSpan(0, 4, {"ph": "test"})  # Only covers "hell"

# RIGHT: Exclusive end
override = OverrideSpan(0, 5, {"ph": "test"})  # Covers "hello"

❌ Assuming Legacy Alignment Works with Duplicates

# WRONG: Legacy alignment cannot handle this correctly
text = "the cat the dog"
overrides = [
    OverrideSpan(0, 3, {"ph": "ðə"}),
    OverrideSpan(8, 11, {"ph": "ði"}),
]
result = phonemize(text, language="en-us", overrides=overrides, alignment="legacy")
# Both overrides apply to first "the" only!

# RIGHT: Use span alignment (default)
result = phonemize(text, language="en-us", overrides=overrides)
# Correctly applies to each "the" instance

Span-Based Phonemization Guide

Overview

Core Types

TokenSpan

OverrideSpan

Exact sub-token language spans

Automatic language routing

Structured stress overrides

PhonemizeResult

Prepared phonemization

Extended Text Layer

SSMD and phrasplit compatibility

Normalize SSMD annotations

Rebase document spans to sentence segments

Character Offset Coordinate System

Basic Rules

Examples

With Markup

Duplicate Words

Alignment Modes

1. Span Alignment (Default)

2. Legacy Word Alignment

Overlap Handling

Snap Mode (Default)

Strict Mode

Language Switching

Phoneme Overrides

Custom Attributes

Best Practices

1. Use Span Alignment (Default)

2. Compute Offsets from Clean Text

3. Use tokenize() for Debugging

4. Check Warnings

5. Test with Duplicates

Common Pitfalls

❌ Using Word Offsets Instead of Character Offsets

❌ Ignoring Clean Text vs Original Text

❌ Forgetting Exclusive End

❌ Assuming Legacy Alignment Works with Duplicates

See Also