Speech effects quality evaluation

Local comparison harness

The no-network harness reads PCM WAV with the Python standard library and writes deterministic 16-bit WAV renders plus CSV and JSON diagnostics:

python scripts/compare_speech_effects.py input.wav comparison-output

It renders the basic phase-vocoder, WSOLA, and ESOLA rate paths, then supported combined WSOLA, ESOLA, and TD-PSOLA pitch/rate cases. CSV/JSON records include runtime, real-time factor, peak, RMS, exact-length error, and continuity diagnostics. Use --rates and --semitones to narrow a run. The harness does not install, invoke, or require Rubber Band, SoundTouch, WORLD, or another external backend.

For envelope A/B renders, add --envelopes. It adds rate transitions from 1.0 to each requested rate over 0.45 output seconds, pitch transitions from 0.0 to each requested semitone target over 0.30 seconds, and combined transitions. For a focused run on a permitted speech WAV:

python scripts/compare_speech_effects.py input.wav comparison-output --rates 0.9 --semitones 2 --envelopes

The generated envelope_*.wav files and metrics record the numeric control points and exact expected frame count. No speech recording is distributed with AudioSig.

The required performance and robustness matrix is reproducible with:

python scripts/benchmark_td_psola.py --output td-psola-benchmark.json

The TD-PSOLA benchmark covers deterministic 1 and 10 second fixtures at 8, 16, 24, and 48 kHz with mono and two-lane inputs, clean/noisy/reverberant material, rates 0.8, 1.0, and 1.25, shifts -4 and +4, runtime, and peak memory. ESOLA’s known-tone ZFR comparison is available with:

python scripts/benchmark_esola.py --synthetic --output esola-benchmark.json

These objective checks cover complete-frame endpoint handling, adaptive trend-window behavior, approximately-two-period TD-PSOLA grains, local-F0 pitch marks, fallback behavior, and exact output contracts. They do not replace the real-speech listening protocol below.

Time-varying envelope evaluation

The envelope regression tests live in tests/test_automation.py, tests/test_speech_envelope.py, and tests/test_speech_envelope_quality.py. They cover analytical integration and inverse-map round trips, exact output sizing, amplitude-coded source landmarks, local pitch trajectories, combined automation, voiced/unvoiced fallback, output-knot jumps, and high-frequency burst guards. These objective tests are diagnostics, not a substitute for listening.

Reproduce the 1, 10, and 60 second envelope benchmark with:

python scripts/benchmark_speech_envelopes.py --output /tmp/speech-envelope-benchmark.json

The script runs static rate, static pitch, static combined, variable-rate, variable-pitch, and combined-variable cases on a deterministic 16 kHz harmonic fixture. Constant-rate baselines are chosen to match the variable-rate output frame count. It records runtime, real-time factor, expected/output frames, length error, finite status, environment, and variable/static runtime ratio. A ratio above 1.0 means the envelope case took longer. This is a single sequential run without warm-up, so treat timings as indicative rather than a performance guarantee.

Recorded on Python 3.13.14, NumPy 2.5.3, AudioSig working-tree metadata 0.1.4, Linux 6.18.33.2 under WSL2, Intel Core Ultra 7 355, 8 logical CPUs. All 18 outputs were finite and had zero frame error. Relative elapsed-time ratios were:

The fixture is synthetic and fully voiced. These timings do not establish perceptual quality or predict every speech input.

The feature is available since AudioSig 0.1.5. After redoing all three A/B comparisons on a permitted user-created PCM WAV, the user could not hear a clear difference and considered the static and envelope renders the same. This is one listener’s subjective result for this source and these settings, not a general claim of equivalence. Findings are in the listening report. The source and rendered files remain local and are not part of release materials.

Variable-rate-only runs use the absolute integrated rate map with WSOLA. Any variable pitch uses TD-PSOLA on voiced regions and mapped WSOLA for unvoiced fallback; the speech path is not formant-preserving and larger shifts can sound more artifact-prone. Results are deterministic within an AudioSig version but are not promised to be bit-identical across versions.

The listening report records the corrected result and can capture further listeners or samples. This listening step is complete for the current sample; overall release acceptance still depends on remaining technical validation.

Listening protocol

For the TD-PSOLA release check, use the corpus categories, randomized A/B or ABX matrix, listener fields, and promotion gates in docs/td-psola-listening-evaluation-2026-07-31.md. No default change is justified by synthetic metrics alone.

Native pitch shifting changes the spectral envelope along with F0 and therefore does not currently preserve vocal formants. Larger shifts are expected to be more artifact-prone; this harness records metrics but does not turn them into a claim of studio-grade perceptual quality.

Deterministic reference metrics and intelligibility validation

Reference metrics are deterministic comparisons of caller-supplied arrays, not perceptual ratings. Pairwise tests cover analytic waveform cases, strict shape and rate rejection, no hidden alignment/transformation, structured result metadata, and synthetic degradation ordering. A seeded voiced synthetic fixture also verifies that stronger added noise lowers both STOI and ESTOI; this is a regression check, not a listening-quality claim.

STOI and ESTOI were gated on a separate, independent black-box numerical check. The golden fixture uses numpy.random.default_rng(20261008) to generate a 40,000-sample standard-normal reference and an estimate equal to that reference plus 0.2 times an independent standard-normal sequence. With both rates at 10 kHz, PySTOI 0.4.1 returned STOI 0.9589928689124777 and ESTOI 0.9558169218900828. AudioSig returned 0.9590914015701738 and 0.9559175969169041, respectively; the test accepts an absolute difference of 2e-4 for each. PySTOI’s upstream Octave-comparison test documents its reference lineage. PySTOI was used only as a temporary external oracle and is not an AudioSig dependency or implementation source.

The NumPy implementations follow the published algorithm descriptions: Taal et al., “A Short-Time Objective Intelligibility Measure for Time-Frequency Weighted Noisy Speech,” ICASSP 2010, doi:10.1109/ICASSP.2010.5495701; and Jensen & Taal, “An Algorithm for Predicting the Intelligibility of Speech Masked by Modulated Noise Maskers,” IEEE/ACM TASLP 2016, doi:10.1109/TASLP.2016.2585878. The independent fixture is broadband noise rather than speech, so it checks numerical agreement only; it does not establish predictive validity or human perceptual quality. AudioSig emits no MOS, loads no neural model, downloads no weights, and never transforms metric inputs implicitly.