API Reference

split_sentences

phrasplit.split_sentences(text: str, language_model: str | None = None, apply_corrections: bool = True, split_on_colon: bool = True, use_spacy: bool | None = None, *, language: str = 'en', model_size: Literal['sm', 'md', 'lg', 'trf'] | None = None) → list[str][source]

Split text into sentences.

By default, selects the highest-quality compatible installed and loadable spaCy model for the requested language, otherwise falls back to regex-based splitting. You can force a specific implementation with use_spacy.

Parameters:

• text – Input text

• language_model – Language model name (e.g., “en_core_web_sm”, “de_core_news_sm”) For spaCy mode: Name of the spaCy model to use For simple mode: Used to determine language for abbreviation handling

• apply_corrections – Whether to apply post-processing corrections for common spaCy errors (URL splitting, abbreviation handling). Default is True. The regex fallback applies the same corrections.

• split_on_colon – Deprecated. Kept for API compatibility (currently unused). spaCy’s default colon behavior is used. Default is True.

• use_spacy – Choose implementation: None (default) selects a compatible local model or uses regex. True requires spaCy and a compatible loadable model. False forces simple regex-based splitting without spaCy.

• language – Language or locale hint used for model selection and abbreviations.

• model_size – Optional exact model tier (sm, md, lg, or trf).

Returns:

List of sentences

Raises:

• SpacyNotInstalledError – If use_spacy=True and spaCy is not installed.

• NoCompatibleSpacyModelError – If forced spaCy has no compatible loadable model.

• ExplicitSpacyModelError – If an explicit model cannot be loaded.

Example:

>>> # Auto-select a compatible local model, otherwise use regex
>>> sentences = split_sentences(text)
>>>
>>> # Force simple mode (even if spaCy is installed)
>>> sentences = split_sentences(text, use_spacy=False)
>>>
>>> # Force spaCy mode (requires a compatible local model)
>>> sentences = split_sentences(text, use_spacy=True)

Note

The simple mode (regex-based) is faster and has no ML dependencies, but is less accurate (~85-90% vs ~95%+ for spaCy) on complex text. For best results with complex text, install spaCy:

pip install phrasplit[nlp]

Example:

from phrasplit import split_sentences

text = "Dr. Smith is here. She has a Ph.D. in Chemistry."
sentences = split_sentences(text)
# ['Dr. Smith is here.', 'She has a Ph.D. in Chemistry.']

# Use simple mode (no spaCy required)
sentences = split_sentences(text, use_spacy=False)

# Automatic highest-available selection, exact overrides, and regex forcing
sentences = split_sentences(text, language="en")
sentences = split_sentences(text, language="de")
sentences = split_sentences(text, language_model="en_core_web_sm")
sentences = split_sentences(text, language="en", model_size="lg")

# split_on_colon is deprecated (kept for compatibility only)
text = "Note: This is important."
sentences = split_sentences(text, split_on_colon=False)

split_clauses

phrasplit.split_clauses(text: str, language_model: str | None = None, use_spacy: bool | None = None, *, language: str = 'en', model_size: Literal['sm', 'md', 'lg', 'trf'] | None = None) → list[str][source]

Split text into comma-separated parts for audiobook creation.

Uses sentence detection, then splits each sentence at commas. The comma stays at the end of each part, creating natural pause points for text-to-speech processing.

Parameters:

• text – Input text

• language_model – Language model name (e.g., “en_core_web_sm”)

• use_spacy – Choose implementation: None (default) selects a compatible local model or uses regex. True requires spaCy and a compatible loadable model. False forces simple regex-based splitting.

• language – Language or locale hint used for model selection and abbreviations.

• model_size – Optional exact model tier; it never falls back to another tier.

Returns:

List of comma-separated parts

Raises:

• SpacyNotInstalledError – If use_spacy=True and spaCy is not installed.

• NoCompatibleSpacyModelError – If forced spaCy has no compatible loadable model.

• ExplicitSpacyModelError – If an explicit model cannot be loaded.

Example:

Input: "I do like coffee, and I like wine."
Output: ["I do like coffee,", "and I like wine."]

Example:

from phrasplit import split_clauses

text = "I like coffee, and I like tea."
clauses = split_clauses(text)
# ['I like coffee,', 'and I like tea.']

# Use simple mode for faster processing
clauses = split_clauses(text, use_spacy=False)

split_paragraphs

phrasplit.split_paragraphs(text: str) → list[str][source]

Split text into paragraphs (separated by double newlines).

Applies preprocessing to fix hyphenated line breaks and normalize whitespace.

Parameters:

text – Input text

Returns:

List of paragraphs (non-empty, stripped)

Example:

from phrasplit import split_paragraphs

text = "First paragraph.\n\nSecond paragraph."
paragraphs = split_paragraphs(text)
# ['First paragraph.', 'Second paragraph.']

split_text

phrasplit.split_text(text: str, mode: str = 'sentence', language_model: str | None = None, apply_corrections: bool = True, split_on_colon: bool = True, use_spacy: bool | None = None, *, language: str = 'en', model_size: Literal['sm', 'md', 'lg', 'trf'] | None = None) → list[Segment][source]

Split text into segments with hierarchical position information.

This function provides a unified interface for text splitting with different granularity levels, while preserving paragraph and sentence structure information. Useful for audiobook generation where different pause lengths are needed between paragraphs vs. sentences vs. clauses.

Parameters:

• text – Input text to split

• mode – Splitting mode. Valid values are "paragraph", "sentence", and "clause".

• language_model – Language model name (e.g., “en_core_web_sm”)

• apply_corrections – Whether to apply post-processing corrections for common spaCy errors (URL splitting, abbreviation handling). Default is True. Applies to sentence/clause modes for both backends.

• split_on_colon – Deprecated. Kept for API compatibility (currently unused). spaCy’s default colon behavior is used. Default is True.

• use_spacy – Choose implementation: None (default) selects a compatible installed/loadable model or regex. True requires spaCy and a compatible loadable model. False forces simple regex-based splitting.

• language – Language or locale hint used for model selection and abbreviations.

• model_size – Optional exact model tier; it never falls back to another tier.

Returns:

• text: The segment text

• paragraph: Paragraph index (0-based)

• sentence: Sentence index within paragraph (0-based). None for paragraph mode.

Return type:

List of Segment namedtuples, each containing

Raises:

• ValueError – If mode is not one of “paragraph”, “sentence”, “clause”

• SpacyNotInstalledError – If use_spacy=True and spaCy is not installed.

• NoCompatibleSpacyModelError – If forced spaCy has no compatible loadable model.

• ExplicitSpacyModelError – If an explicit model cannot be loaded.

Example:

>>> segments = split_text("Hello world. How are you?\n\nNew paragraph.")
>>> for seg in segments:
...     print(f"P{seg.paragraph} S{seg.sentence}: {seg.text}")
P0 S0: Hello world.
P0 S1: How are you?
P1 S0: New paragraph.

>>> # Detect paragraph changes for longer pauses
>>> for i, seg in enumerate(segments):
...     if i > 0 and seg.paragraph != segments[i-1].paragraph:
...         print("--- paragraph break ---")
...     print(seg.text)

Example:

from phrasplit import split_text, Segment

text = "First sentence. Second sentence.\n\nNew paragraph."
segments = split_text(text, mode="sentence")

for seg in segments:
    print(f"P{seg.paragraph} S{seg.sentence}: {seg.text}")
# P0 S0: First sentence.
# P0 S1: Second sentence.
# P1 S0: New paragraph.

# Clause mode for finer granularity
text = "Hello, world.\n\nGoodbye, friend."
segments = split_text(text, mode="clause")
# Returns clauses with paragraph and sentence indices

# Use simple mode (no spaCy)
segments = split_text(text, mode="sentence", use_spacy=False)

split_long_lines

phrasplit.split_long_lines(text: str, max_length: int, language_model: str | None = None, use_spacy: bool | None = None, *, language: str = 'en', model_size: Literal['sm', 'md', 'lg', 'trf'] | None = None) → list[str][source]

Split lines exceeding max_length at clause/sentence boundaries.

Strategy: 1. First try to split at sentence boundaries 2. If still too long, split at clause boundaries (commas, semicolons, etc.) 3. If still too long, split at word boundaries

Parameters:

• text – Input text

• max_length – Maximum line length in characters (must be positive)

• language_model – Language model name (e.g., “en_core_web_sm”)

• use_spacy – Choose implementation: None (default) selects a compatible local model or uses regex. True requires spaCy and a compatible loadable model. False forces simple regex-based splitting.

• language – Language or locale hint used for model selection and abbreviations.

• model_size – Optional exact model tier; it never falls back to another tier.

Returns:

List of lines, each within max_length (except single words exceeding limit)

Raises:

• ValueError – If max_length is less than 1

• SpacyNotInstalledError – If use_spacy=True and spaCy is not installed.

• NoCompatibleSpacyModelError – If forced spaCy has no compatible loadable model.

• ExplicitSpacyModelError – If an explicit model cannot be loaded.

Example:

from phrasplit import split_long_lines

text = "This is a very long sentence that needs to be split into smaller parts."
lines = split_long_lines(text, max_length=40)

# Use simple mode
lines = split_long_lines(text, max_length=40, use_spacy=False)

Offset APIs

phrasplit.split_with_offsets(text: str, *, mode: str = 'sentence', use_spacy: bool | None = None, language_model: str | None = None, language: str = 'en', model_size: Literal['sm', 'md', 'lg', 'trf'] | None = None, apply_corrections: bool = True, max_chars: int | None = None, inline_markup: bool = False) → list[SplitSegment][source]

Split text into segments with character offsets and stable IDs.

This is the main API for offset-preserving segmentation, designed for downstream processing where exact character positions are critical.

Exact-Slice Policy

This function implements the exact-slice policy. For every returned segment, the following invariant ALWAYS holds:

segment.text == text[segment.char_start:segment.char_end]

This guarantee means:

• Offsets map precisely to the original input text

• No whitespace normalization or stripping breaks the mapping

• Downstream code can reliably use offsets for span slicing

• Integration with token alignment and markup slicing is safe

Key features:

• Returns segments with precise character offsets (char_start, char_end)

• Generates stable, hierarchical IDs (e.g., “p0s1”, “p0s2c3”)

• Maintains exact-slice invariant in all modes

• Supports both spaCy (accurate) and regex (fast) backends

• Optional max_chars safety splitting with deterministic boundaries

Parameters:

• text – Input text to split

• mode – Splitting granularity. Valid values are "paragraph", "sentence", and "clause".

• use_spacy – Backend selection. None (default) selects a compatible installed/loadable local model or falls back to regex; True forces spaCy; False forces regex-based splitting.

• language_model – Language model name (e.g., “en_core_web_sm”, “de_core_news_sm”) Used for both spaCy model selection and abbreviation handling

• language – Language or locale hint used for model selection and abbreviations.

• model_size – Optional exact model tier; it never falls back to another tier.

• apply_corrections – Whether to apply post-processing corrections for common spaCy errors (URL splitting, abbreviation handling, ellipsis). Default is True. Only applies to spaCy mode.

• max_chars – Optional maximum segment length. Segments exceeding this will be split further at whitespace/punctuation boundaries while maintaining the exact-slice invariant. Split segments get IDs like “p0s1:m0”, “p0s1:m1”, etc.

• inline_markup – Opt-in inline XHTML markup handling (default False). When True, the regex backend keeps inline tags balanced across sentence boundaries: closing tags after punctuation stay in the previous segment and opening tags after whitespace start the next segment. Requires the regex backend; passing use_spacy=True with inline_markup=True raises ValueError. Default behavior (plain text) is unchanged when False.

Returns:

• id: Stable identifier (e.g., “p0s1c2” or “p0s1:m0”)

• text: Segment text content (exact slice of input)

• char_start, char_end: Character offsets in original text

• paragraph_idx, sentence_idx, clause_idx: Hierarchical indices

• meta: Additional metadata (method, mode, etc.)

Return type:

List of SplitSegment objects, each containing

Raises:

• ValueError – If mode is invalid or max_chars < 1

• SpacyNotInstalledError – If use_spacy=True and spaCy is not installed.

• NoCompatibleSpacyModelError – If forced spaCy has no compatible loadable model.

• ExplicitSpacyModelError – If an explicit model cannot be loaded.

Example:

>>> text = "Hello world. How are you?\n\nNew paragraph."
>>> segments = split_with_offsets(text, mode="sentence")
>>> for seg in segments:
...     # Verify exact-slice invariant
...     assert text[seg.char_start:seg.char_end] == seg.text
...     print(f"{seg.id}: {seg.text!r}")
p0s0: 'Hello world.'
p0s1: 'How are you?'
p1s0: 'New paragraph.'

>>> # With max_chars safety splitting
>>> long_text = "word " * 100
>>> segments = split_with_offsets(long_text, max_chars=50)
>>> all(len(seg.text) <= 50 for seg in segments)
True
>>> # Exact-slice invariant still holds
>>> all(long_text[s.char_start:s.char_end] == s.text for s in segments)
True

Note

• Segments may include leading/trailing whitespace from the original text

• IDs are stable and deterministic across runs with same input and settings

• For SSMD/markup integration, offsets are in the coordinate space of the input text (before or after escaping, depending on your workflow)

phrasplit.iter_split_with_offsets(text: str, *, mode: str = 'sentence', use_spacy: bool | None = None, language_model: str | None = None, language: str = 'en', model_size: Literal['sm', 'md', 'lg', 'trf'] | None = None, apply_corrections: bool = True, max_chars: int | None = None, inline_markup: bool = False) → Iterator[SplitSegment][source]

Iterator facade over split_with_offsets().

The current implementation computes the complete segment list before yielding. It is useful for iterator-oriented integrations, but it does not yet provide incremental parsing, bounded memory, or earlier time-to-first-segment behavior.

Parameters:

• text – Input text to split

• mode – Splitting granularity (“paragraph”, “sentence”, or “clause”)

• use_spacy – Backend selection (None=auto, True=spaCy, False=regex)

• language_model – Language model name for NLP/abbreviations

• language – Language or locale hint used for model selection and abbreviations.

• model_size – Optional exact model tier.

• apply_corrections – Whether to apply post-processing corrections for common spaCy errors (URL splitting, abbreviation handling, ellipsis). Default is True. Only applies to spaCy mode.

• max_chars – Optional maximum segment length

Yields:

SplitSegment objects in document order

Example:

>>> text = "First sentence. Second sentence.\n\nNew paragraph."
>>> for segment in iter_split_with_offsets(text, mode="sentence"):
...     print(f"{segment.id}: {segment.text}")
p0s0: First sentence.
p0s1: Second sentence.
p1s0: New paragraph.

Note

• Segments are yielded in document order

• Same offset guarantees as split_with_offsets()

• The underlying splitter may use its documented model cache/global diagnostic state; callers must not interpret this facade as a thread-isolated stream.

Both APIs preserve segment.text == text[segment.char_start:segment.char_end]. inline_markup=True is available only with the regex backend and raises ValueError when explicitly combined with spaCy. The iterator is currently a facade over the list API and does not claim incremental or bounded-memory processing.

Segment

class phrasplit.Segment(text: str, paragraph: int, sentence: int | None = None)[source]

Bases: NamedTuple

A text segment with position information.

- ``text``

The text content of the segment.

- ``paragraph``

Paragraph index (0-based) within the document.

- ``sentence``

Sentence index (0-based) within the paragraph. None for paragraph mode.

text: str

Alias for field number 0

paragraph: int

Alias for field number 1

sentence: int | None

Alias for field number 2

A named tuple representing a text segment with position information.

Fields:

• text (str): The text content of the segment

• paragraph (int): Paragraph index (0-based) within the document

• sentence (int | None): Sentence index (0-based) within the paragraph. None for paragraph mode.

Example:

from phrasplit import split_text, Segment

segments = split_text("Hello world.", mode="sentence")
seg = segments[0]

# Access by name
print(seg.text)       # "Hello world."
print(seg.paragraph)  # 0
print(seg.sentence)   # 0

# Access by index
print(seg[0])  # "Hello world."
print(seg[1])  # 0
print(seg[2])  # 0

# Unpack
text, para, sent = seg

splitter module

Text splitting utilities using spaCy for NLP-based sentence and clause detection.

This module provides two implementations: 1. spaCy-based (default when available): High accuracy, handles complex cases 2. Regex-based (fallback): Faster, simpler, good for common cases

The implementation is selected automatically based on spaCy availability, or can be controlled via the use_spacy parameter.

class phrasplit.splitter.Segment(text: str, paragraph: int, sentence: int | None = None)[source]

Bases: NamedTuple

A text segment with position information.

- ``text``

The text content of the segment.

- ``paragraph``

Paragraph index (0-based) within the document.

- ``sentence``

Sentence index (0-based) within the paragraph. None for paragraph mode.

text: str

Alias for field number 0

paragraph: int

Alias for field number 1

sentence: int | None

Alias for field number 2

phrasplit.splitter.split_paragraphs(text: str) → list[str][source]

Split text into paragraphs (separated by double newlines).

Applies preprocessing to fix hyphenated line breaks and normalize whitespace.

Parameters:

text – Input text

Returns:

List of paragraphs (non-empty, stripped)

phrasplit.splitter.split_sentences(text: str, language_model: str | None = None, apply_corrections: bool = True, split_on_colon: bool = True, use_spacy: bool | None = None, *, language: str = 'en', model_size: Literal['sm', 'md', 'lg', 'trf'] | None = None) → list[str][source]

Split text into sentences.

By default, selects the highest-quality compatible installed and loadable spaCy model for the requested language, otherwise falls back to regex-based splitting. You can force a specific implementation with use_spacy.

Parameters:

• text – Input text

• language_model – Language model name (e.g., “en_core_web_sm”, “de_core_news_sm”) For spaCy mode: Name of the spaCy model to use For simple mode: Used to determine language for abbreviation handling

• apply_corrections – Whether to apply post-processing corrections for common spaCy errors (URL splitting, abbreviation handling). Default is True. The regex fallback applies the same corrections.

• split_on_colon – Deprecated. Kept for API compatibility (currently unused). spaCy’s default colon behavior is used. Default is True.

• use_spacy – Choose implementation: None (default) selects a compatible local model or uses regex. True requires spaCy and a compatible loadable model. False forces simple regex-based splitting without spaCy.

• language – Language or locale hint used for model selection and abbreviations.

• model_size – Optional exact model tier (sm, md, lg, or trf).

Returns:

List of sentences

Raises:

• SpacyNotInstalledError – If use_spacy=True and spaCy is not installed.

• NoCompatibleSpacyModelError – If forced spaCy has no compatible loadable model.

• ExplicitSpacyModelError – If an explicit model cannot be loaded.

Example:

>>> # Auto-select a compatible local model, otherwise use regex
>>> sentences = split_sentences(text)
>>>
>>> # Force simple mode (even if spaCy is installed)
>>> sentences = split_sentences(text, use_spacy=False)
>>>
>>> # Force spaCy mode (requires a compatible local model)
>>> sentences = split_sentences(text, use_spacy=True)

Note

The simple mode (regex-based) is faster and has no ML dependencies, but is less accurate (~85-90% vs ~95%+ for spaCy) on complex text. For best results with complex text, install spaCy:

pip install phrasplit[nlp]

phrasplit.splitter.split_clauses(text: str, language_model: str | None = None, use_spacy: bool | None = None, *, language: str = 'en', model_size: Literal['sm', 'md', 'lg', 'trf'] | None = None) → list[str][source]

Split text into comma-separated parts for audiobook creation.

Uses sentence detection, then splits each sentence at commas. The comma stays at the end of each part, creating natural pause points for text-to-speech processing.

Parameters:

• text – Input text

• language_model – Language model name (e.g., “en_core_web_sm”)

• use_spacy – Choose implementation: None (default) selects a compatible local model or uses regex. True requires spaCy and a compatible loadable model. False forces simple regex-based splitting.

• language – Language or locale hint used for model selection and abbreviations.

• model_size – Optional exact model tier; it never falls back to another tier.

Returns:

List of comma-separated parts

Raises:

• SpacyNotInstalledError – If use_spacy=True and spaCy is not installed.

• NoCompatibleSpacyModelError – If forced spaCy has no compatible loadable model.

• ExplicitSpacyModelError – If an explicit model cannot be loaded.

Example:

Input: "I do like coffee, and I like wine."
Output: ["I do like coffee,", "and I like wine."]

phrasplit.splitter.split_long_lines(text: str, max_length: int, language_model: str | None = None, use_spacy: bool | None = None, *, language: str = 'en', model_size: Literal['sm', 'md', 'lg', 'trf'] | None = None) → list[str][source]

Split lines exceeding max_length at clause/sentence boundaries.

Strategy: 1. First try to split at sentence boundaries 2. If still too long, split at clause boundaries (commas, semicolons, etc.) 3. If still too long, split at word boundaries

Parameters:

• text – Input text

• max_length – Maximum line length in characters (must be positive)

• language_model – Language model name (e.g., “en_core_web_sm”)

• use_spacy – Choose implementation: None (default) selects a compatible local model or uses regex. True requires spaCy and a compatible loadable model. False forces simple regex-based splitting.

• language – Language or locale hint used for model selection and abbreviations.

• model_size – Optional exact model tier; it never falls back to another tier.

Returns:

List of lines, each within max_length (except single words exceeding limit)

Raises:

• ValueError – If max_length is less than 1

• SpacyNotInstalledError – If use_spacy=True and spaCy is not installed.

• NoCompatibleSpacyModelError – If forced spaCy has no compatible loadable model.

• ExplicitSpacyModelError – If an explicit model cannot be loaded.

phrasplit.splitter.split_text(text: str, mode: str = 'sentence', language_model: str | None = None, apply_corrections: bool = True, split_on_colon: bool = True, use_spacy: bool | None = None, *, language: str = 'en', model_size: Literal['sm', 'md', 'lg', 'trf'] | None = None) → list[Segment][source]

Split text into segments with hierarchical position information.

This function provides a unified interface for text splitting with different granularity levels, while preserving paragraph and sentence structure information. Useful for audiobook generation where different pause lengths are needed between paragraphs vs. sentences vs. clauses.

Parameters:

• text – Input text to split

• mode – Splitting mode. Valid values are "paragraph", "sentence", and "clause".

• language_model – Language model name (e.g., “en_core_web_sm”)

• apply_corrections – Whether to apply post-processing corrections for common spaCy errors (URL splitting, abbreviation handling). Default is True. Applies to sentence/clause modes for both backends.

• split_on_colon – Deprecated. Kept for API compatibility (currently unused). spaCy’s default colon behavior is used. Default is True.

• use_spacy – Choose implementation: None (default) selects a compatible installed/loadable model or regex. True requires spaCy and a compatible loadable model. False forces simple regex-based splitting.

• language – Language or locale hint used for model selection and abbreviations.

• model_size – Optional exact model tier; it never falls back to another tier.

Returns:

• text: The segment text

• paragraph: Paragraph index (0-based)

• sentence: Sentence index within paragraph (0-based). None for paragraph mode.

Return type:

List of Segment namedtuples, each containing

Raises:

• ValueError – If mode is not one of “paragraph”, “sentence”, “clause”

• SpacyNotInstalledError – If use_spacy=True and spaCy is not installed.

• NoCompatibleSpacyModelError – If forced spaCy has no compatible loadable model.

• ExplicitSpacyModelError – If an explicit model cannot be loaded.

Example:

>>> segments = split_text("Hello world. How are you?\n\nNew paragraph.")
>>> for seg in segments:
...     print(f"P{seg.paragraph} S{seg.sentence}: {seg.text}")
P0 S0: Hello world.
P0 S1: How are you?
P1 S0: New paragraph.

>>> # Detect paragraph changes for longer pauses
>>> for i, seg in enumerate(segments):
...     if i > 0 and seg.paragraph != segments[i-1].paragraph:
...         print("--- paragraph break ---")
...     print(seg.text)

phrasplit.splitter.split_with_offsets(text: str, *, mode: str = 'sentence', use_spacy: bool | None = None, language_model: str | None = None, language: str = 'en', model_size: Literal['sm', 'md', 'lg', 'trf'] | None = None, apply_corrections: bool = True, max_chars: int | None = None, inline_markup: bool = False) → list[SplitSegment][source]

Split text into segments with character offsets and stable IDs.

This is the main API for offset-preserving segmentation, designed for downstream processing where exact character positions are critical.

Exact-Slice Policy

This function implements the exact-slice policy. For every returned segment, the following invariant ALWAYS holds:

segment.text == text[segment.char_start:segment.char_end]

This guarantee means:

• Offsets map precisely to the original input text

• No whitespace normalization or stripping breaks the mapping

• Downstream code can reliably use offsets for span slicing

• Integration with token alignment and markup slicing is safe

Key features:

• Returns segments with precise character offsets (char_start, char_end)

• Generates stable, hierarchical IDs (e.g., “p0s1”, “p0s2c3”)

• Maintains exact-slice invariant in all modes

• Supports both spaCy (accurate) and regex (fast) backends

• Optional max_chars safety splitting with deterministic boundaries

Parameters:

• text – Input text to split

• mode – Splitting granularity. Valid values are "paragraph", "sentence", and "clause".

• use_spacy – Backend selection. None (default) selects a compatible installed/loadable local model or falls back to regex; True forces spaCy; False forces regex-based splitting.

• language_model – Language model name (e.g., “en_core_web_sm”, “de_core_news_sm”) Used for both spaCy model selection and abbreviation handling

• language – Language or locale hint used for model selection and abbreviations.

• model_size – Optional exact model tier; it never falls back to another tier.

• apply_corrections – Whether to apply post-processing corrections for common spaCy errors (URL splitting, abbreviation handling, ellipsis). Default is True. Only applies to spaCy mode.

• max_chars – Optional maximum segment length. Segments exceeding this will be split further at whitespace/punctuation boundaries while maintaining the exact-slice invariant. Split segments get IDs like “p0s1:m0”, “p0s1:m1”, etc.

• inline_markup – Opt-in inline XHTML markup handling (default False). When True, the regex backend keeps inline tags balanced across sentence boundaries: closing tags after punctuation stay in the previous segment and opening tags after whitespace start the next segment. Requires the regex backend; passing use_spacy=True with inline_markup=True raises ValueError. Default behavior (plain text) is unchanged when False.

Returns:

• id: Stable identifier (e.g., “p0s1c2” or “p0s1:m0”)

• text: Segment text content (exact slice of input)

• char_start, char_end: Character offsets in original text

• paragraph_idx, sentence_idx, clause_idx: Hierarchical indices

• meta: Additional metadata (method, mode, etc.)

Return type:

List of SplitSegment objects, each containing

Raises:

• ValueError – If mode is invalid or max_chars < 1

• SpacyNotInstalledError – If use_spacy=True and spaCy is not installed.

• NoCompatibleSpacyModelError – If forced spaCy has no compatible loadable model.

• ExplicitSpacyModelError – If an explicit model cannot be loaded.

Example:

>>> text = "Hello world. How are you?\n\nNew paragraph."
>>> segments = split_with_offsets(text, mode="sentence")
>>> for seg in segments:
...     # Verify exact-slice invariant
...     assert text[seg.char_start:seg.char_end] == seg.text
...     print(f"{seg.id}: {seg.text!r}")
p0s0: 'Hello world.'
p0s1: 'How are you?'
p1s0: 'New paragraph.'

>>> # With max_chars safety splitting
>>> long_text = "word " * 100
>>> segments = split_with_offsets(long_text, max_chars=50)
>>> all(len(seg.text) <= 50 for seg in segments)
True
>>> # Exact-slice invariant still holds
>>> all(long_text[s.char_start:s.char_end] == s.text for s in segments)
True

Note

• Segments may include leading/trailing whitespace from the original text

• IDs are stable and deterministic across runs with same input and settings

• For SSMD/markup integration, offsets are in the coordinate space of the input text (before or after escaping, depending on your workflow)

phrasplit.splitter.iter_split_with_offsets(text: str, *, mode: str = 'sentence', use_spacy: bool | None = None, language_model: str | None = None, language: str = 'en', model_size: Literal['sm', 'md', 'lg', 'trf'] | None = None, apply_corrections: bool = True, max_chars: int | None = None, inline_markup: bool = False) → Iterator[SplitSegment][source]

Iterator facade over split_with_offsets().

The current implementation computes the complete segment list before yielding. It is useful for iterator-oriented integrations, but it does not yet provide incremental parsing, bounded memory, or earlier time-to-first-segment behavior.

Parameters:

• text – Input text to split

• mode – Splitting granularity (“paragraph”, “sentence”, or “clause”)

• use_spacy – Backend selection (None=auto, True=spaCy, False=regex)

• language_model – Language model name for NLP/abbreviations

• language – Language or locale hint used for model selection and abbreviations.

• model_size – Optional exact model tier.

• apply_corrections – Whether to apply post-processing corrections for common spaCy errors (URL splitting, abbreviation handling, ellipsis). Default is True. Only applies to spaCy mode.

• max_chars – Optional maximum segment length

Yields:

SplitSegment objects in document order

Example:

>>> text = "First sentence. Second sentence.\n\nNew paragraph."
>>> for segment in iter_split_with_offsets(text, mode="sentence"):
...     print(f"{segment.id}: {segment.text}")
p0s0: First sentence.
p0s1: Second sentence.
p1s0: New paragraph.

Note

• Segments are yielded in document order

• Same offset guarantees as split_with_offsets()

• The underlying splitter may use its documented model cache/global diagnostic state; callers must not interpret this facade as a thread-isolated stream.