PolyNorm diagnostic benchmark

Spokenform’s PolyNorm adapter evaluates the overlap between the external Apple PolyNorm-Bench corpus and the seven base languages supported by Spokenform. The overlap is de-DE, en-US, es-MX, fr-FR, and it-IT;

This benchmark remains limited to its existing seven-language overlap. Japanese, Korean, and Chinese runtime support is covered by focused spokenform regression tests, not by this PolyNorm parity set. the default run derives its case count from the selected cached files because upstream locale files may differ in length. PolyNorm BCP-47 tags are translated to regional Spokenform identifiers only inside the benchmark adapter; they are not a Spokenform API convention.

The source is pinned to dataset commit f3c67e047bea6b7c40bc2466c0fdaad51d8ce67d in apple/ml-speech-polynorm-bench. The dataset is CC BY-NC-ND 4.0. The first download requires an explicit license acknowledgement:

python -m benchmarks.polynorm --accept-license

After downloading, offline and focused runs are available:

python -m benchmarks.polynorm --offline
python -m benchmarks.polynorm --offline --locale en-US --category Date
python -m benchmarks.polynorm --offline --case en-US:1
python -m benchmarks.polynorm --offline --show-failures all
python -m benchmarks.polynorm --offline --speech-wer-threshold 0.5
python -m benchmarks.polynorm --offline --report html

Use --speech-wer-threshold VALUE to persist only failure entries whose word error rate is strictly greater than VALUE. The option filters failures.jsonl and failures.md only; summary metrics and evaluated-case counts still cover the complete run. With no threshold, all current failure entries are stored.

The overlap files are not assumed to have equal lengths; the evaluator derives the case count from the cached JSONL files. Reports therefore distinguish raw metrics, canonicalized category metrics, ownership-aware metrics, and a local reviewed subset that excludes annotations in a separate quarantine file. The upstream category and ground-truth text are never rewritten.

Cached JSONL data and the upstream license live under .cache/polynorm-bench/<commit>/. Reports live under benchmark-results/polynorm/<run-id>/; only metrics and metadata are intended for summary consumption, while text-bearing failure reports are local. The benchmark records dataset/source/dependency environment fingerprints, literal exactness, raw speech exactness, locale-aware speech-equivalent exactness, presentation-only disagreements, semantic failures, word error rate, unchanged cases, exceptions, residual source-symbol counts, locale/category/ownership aggregates, locale × canonical-category metrics, and per-failure changed stages/source rules. Each failure also records the primary rule, claim owner, failure phase, winning source span, protection reason, numeric speech policy, and render mode. gate_metrics keeps safety, owned, extended-candidate, protected, and per-locale views separate so protected or unsupported categories are not mistaken for regressions in owned behavior. Raw speech exactness remains the PolyNorm-comparable metric; equivalent speech is a parallel diagnostic. It continues after mismatches and individual runtime exceptions and does not add a score threshold to normal CI. Questionable upstream rows remain visible with quarantine reasons and are excluded only from reviewed metrics.

Each run now also writes a self-contained report.html by default. The local dashboard includes KPI cards, a locale table, a canonical-category table, a locale × category view, a diagnostics/gate section, a filterable failure explorer, metadata, and an optional oracle section when --candidate-oracle data is present. Use --report none to skip HTML generation while keeping the existing JSONL, Markdown, and summary artifacts unchanged.

The local quarantine annotations are intentionally limited to eight upstream rows whose expected text is malformed, questionable, or inconsistent with the source (for example, an instruction instead of normalized speech or an inconsistent coordinate). They are never rewritten. German compact currency is an explicit policy: the major unit and minor numeric value are spoken without und or Cent; locale punctuation is selected before separator heuristics; URLs, e-mail addresses, versions, and caller-protected spans remain literal by default.