buchwandler

Advanced Topics

Choosing a time-stretch backend

AudioSig has three explicit time-stretch backends. The basic phase vocoder is a generic numerical reference path; it uses STFT/ISTFT phase propagation and can sound phasey around transients. WSOLA is the speech-oriented default for apply_speech_effects and searches for waveform-similar splice points in the time domain. ESOLA is an experimental speech backend that aligns fixed waveform frames using detected epochs; it supports 0.5 <= rate <= 2.0 and requires sample_rate. WSOLA, ESOLA, and TD-PSOLA process every non-sample dimension as an independent lane, so they do not guarantee a stereo image or shared pitch/epoch decisions. For stereo speech where image coherence matters, provide a shared analysis signal and apply its decisions in an application-level channel wrapper; ordinary batch and arbitrary-axis processing remains supported.

Use WSOLA when processing speech:

from audiosig import apply_speech_effects, time_stretch

stretched = time_stretch(
    audio,
    rate=1.2,
    sample_rate=24_000,
    method="wsola",
)
speech = apply_speech_effects(audio, sample_rate=24_000, rate=1.2)

Use ESOLA for an experimental moderate-range speech comparison:

stretched = time_stretch(
    audio,
    rate=1.25,
    sample_rate=24_000,
    method="esola",
)

Use the basic phase vocoder for generic array processing or controlled comparisons:

from audiosig import time_stretch

# Control FFT resolution
result = time_stretch(
    audio,
    rate=1.2,
    method="phase_vocoder",
    n_fft=4096,        # Higher = better frequency resolution, worse time resolution
    hop_length=1024,   # Controls overlap between frames
)

Backend trade-offs

  • WSOLA frame/overlap/search: Defaults are 30/10/10 ms and are sample-rate aware; larger searches cost more and can select less stable matches.

  • Phase-vocoder n_fft: Larger values improve frequency separation but worsen time precision.

  • Phase-vocoder hop_length: Typically n_fft // 4 for the basic backend.

  • ESOLA frame/epoch alignment: Uses a fixed 20 ms frame, 50% synthesis overlap, NumPy epoch extraction, and a raised-cosine pairwise crossfade. It is speech-specific and remains experimental until real-speech listening results establish a stable benefit over WSOLA.

The standard compositor plans pitch and rate together using tsm_rate = rate / pitch_ratio, then resamples once when necessary. Selecting method="td_psola" uses a separate direct pitch/prosody path instead: it tracks voicing and pulses, changes synthesis-pulse spacing for voiced speech, and uses WSOLA for unvoiced duration fallback. Its conservative limits are -6..+6 semitones and 0.75..1.5 rate. If too few pitch marks are available, the direct path explicitly falls back. Pure pitch requests use WSOLA plus resampling for the requested ratio; duration-modification requests preserve unvoiced material with duration-only WSOLA rather than globally tonalizing it. The backend does not silently return unchanged audio. All paths preserve exact output-length and dtype contracts, but TD-PSOLA does not guarantee formant preservation and should not be treated as a general music or polyphonic pitch-shifting backend.

TD-PSOLA remains experimental until objective, runtime, and real-speech listening gates pass. See the evaluation protocol.

Time-varying speech envelopes

apply_speech_effects_envelope is available since AudioSig 0.1.5. It accepts only numeric control points and leaves higher-level meaning to calling applications. Rate values are positive playback factors; pitch values are semitones. Control-point times use output seconds, both curves interpolate linearly, the first point is at zero, and the final point is held. No repeated calls to static effects or chunk stitching are used.

For output time y, the rate map consumes source time x(y) = integral(0, y, r(u) du). Linear segments are integrated analytically in float64. AudioSig solves the inverse map at the input duration and rounds output_seconds * sample_rate once to determine the output frame count. speech_effects_output_frames exposes that same calculation for downstream timing prediction. Pitch-only envelopes preserve exact input length.

Variable rate-only processing defaults to mapped WSOLA, where each expected source frame is computed from the absolute integrated map and overlap search remains local and deterministic. Selecting method="td_psola" uses voiced pulse synthesis with mapped WSOLA fallback for rate-only curves. When pitch varies, TD-PSOLA controls voiced pulse spacing from output-time semitone values, while unvoiced spans use the same mapped WSOLA rate trajectory. This coordinated path is speech-oriented and does not claim vocal-formant preservation.

Omitted rate and pitch curves mean 1.0 and 0.0. Clips shorter than the requested transition duration evaluate only the output-time portion that exists; they do not compress the curve. Digital silence remains digital silence. The same timing map and control points apply to all channels. Identical inputs and parameters are deterministic within an AudioSig version, but bit-identical output across versions is not promised.

Resampling Internals

AudioSig uses windowed-sinc interpolation:

from audiosig import resample

# Control filter characteristics
result = resample(
    audio,
    source_rate=48000,
    target_rate=16000,
    filter_width=64,     # Wider = sharper cutoff, slower
    rolloff=0.945,       # Anti-aliasing filter rolloff
    length_mode='round', # Output length calculation
)

Length Modes

  • 'round': AudioSig’s default, matches historical behavior

  • 'ceil': Matches librosa’s output length semantics

from audiosig import resample

# Compare length modes
result_round = resample(audio, source_rate=44100, target_rate=16000, length_mode='round')
result_ceil = resample(audio, source_rate=44100, target_rate=16000, length_mode='ceil')

print(f"Round: {len(result_round)}")
print(f"Ceil: {len(result_ceil)}")

Loudness measurement

Loudness and peak metrics are deliberately separate from amplitude transforms. peak_normalize sets the largest discrete sample, RMS/short-time energy describes a chosen window, integrated loudness reports gated BS.1770-style LUFS, and true peak estimates inter-sample dBTP. Equal sample peaks do not imply equal perceived loudness.

The v1 meter accepts finite one-dimensional mono arrays and uses float64 calculation paths. It analyzes complete 400 ms blocks with a 100 ms hop, applies K-weighting, then uses the -70 LUFS absolute gate and -10 LU relative gate. It returns -math.inf for silence and for audio shorter than one complete block; it never pads, clips, limits, or automatically normalizes. The default true-peak oversampling factor is 4, configurable for callers that need a different cost/accuracy trade-off.

import audiosig

metrics = audiosig.measure_loudness(audio, sample_rate=24_000)
print(metrics.integrated_lufs)  # LUFS
print(metrics.sample_peak_dbfs)  # dBFS
print(metrics.true_peak_dbtp)   # dBTP

The stable release contract for downstream PyKokoro code is audiosig>=0.1.3 for LoudnessMetrics, integrated_loudness, sample_peak_dbfs, true_peak_dbtp, and measure_loudness, plus the existing apply_gain_db. AudioSig supplies measurements and generic gain only; target LUFS and voice calibration remain application policy.

VAD Algorithm Details

Normalized Energy VAD

The normalized energy VAD works by:

  1. Framing the audio into non-overlapping segments

  2. Computing RMS energy per frame

  3. Normalizing to [0, 1] range

  4. Thresholding to detect activity

from audiosig import normalized_energy_vad

# The algorithm in detail
activity = normalized_energy_vad(
    audio,
    sample_rate=24000,
    frame_duration_ms=10,     # 10ms frames = 240 samples
    energy_threshold=0.15,    # 15% of max energy
    pad_end=False,            # Discard trailing partial frame
)

Relative dB VAD

The relative dB VAD works by:

  1. Computing short-time energy

  2. Finding peak energy

  3. Setting threshold relative to peak

  4. Detecting frames above threshold

from audiosig import relative_db_vad

activity = relative_db_vad(
    audio,
    frame_length=2048,
    hop_length=512,
    threshold_db=40.0,  # 40 dB below peak
)

Smooth Cut-Point Selection

find_smooth_cut_point is a bounded, sample-domain primitive for applications that already know a legal interval. It uses a cumulative-sum local RMS calculation plus endpoint amplitude, cross-boundary slope, and anchor-distance costs. The implementation allocates work proportional to the local search span, not candidate count times window length, and conservatively aggregates all non-sample lanes.

The function accepts float32 or float64 NumPy arrays, supports an explicit sample axis, never mutates input, and returns a legal index even when no quiet run exists. An empty array returns None only for the explicit start=0, end=0 contract. AudioShapeError and InvalidParameterError report invalid arrays and parameters.

Keep semantic decisions outside AudioSig: callers choose the legal interval, interpret timestamps or application metadata, and decide whether to retry. Scoring weights are intentionally internal and are not public tuning parameters.

Edge Cases and Special Handling

Empty Audio

import numpy as np
from audiosig import trim, split

# Empty arrays are handled gracefully
empty = np.array([], dtype=np.float32)

# trim returns empty array and [0, 0]
trimmed, interval = trim(empty)
assert len(trimmed) == 0
assert np.array_equal(interval, [0, 0])

# split returns empty intervals
intervals = split(empty)
assert len(intervals) == 0

All public transformations, including time_stretch, pitch_shift, gain, normalization, and resampling, accept an empty sample axis after validating their parameters and return an empty copy with the original shape and dtype.

Silent Audio

import numpy as np
from audiosig import trim, peak_normalize

# All-zeros audio
silence = np.zeros(24000, dtype=np.float32)

# With the default peak-relative reference, uniform silence remains unchanged.
trimmed, interval = trim(silence, top_db=40.0, ref=np.max)
assert len(trimmed) == 24000
assert np.array_equal(interval, [0, 24000])

# A fixed nonzero reference classifies the all-zero signal as silence.
trimmed, interval = trim(silence, top_db=40.0, ref=1.0)
assert len(trimmed) == 0
assert np.array_equal(interval, [0, 0])

# Normalization preserves silence
normalized = peak_normalize(silence)
assert np.all(normalized == 0)

Very Short Audio

import numpy as np
from audiosig import time_stretch, pitch_shift

# Single sample
single = np.array([0.5], dtype=np.float32)

# Time stretching works with minimum length 1
result = time_stretch(single, rate=2.0)
assert len(result) >= 1

# Pitch shifting preserves length
result = pitch_shift(single, sample_rate=24000, semitones=3.0)
assert len(result) == 1

Multi-Channel and Batch Processing

Stereo Audio

import numpy as np
from audiosig import time_stretch, pitch_shift

# Stereo audio: (2, samples)
stereo = np.random.randn(2, 24000).astype(np.float32)

# Processing works on last axis by default
stretched = time_stretch(stereo, rate=1.1)
assert stretched.shape[0] == 2  # Channels preserved

# Can specify axis explicitly
result = time_stretch(stereo, rate=1.1, axis=-1)

Batch Dimensions

import numpy as np
from audiosig import time_stretch

# Batch of audio: (batch, channels, samples)
batch = np.random.randn(8, 2, 24000).astype(np.float32)

# Processing preserves batch dimensions
result = time_stretch(batch, rate=1.2)
assert result.shape == (8, 2, 20000)  # Shorter due to rate > 1

Error Handling Strategies

Fail-Fast vs Fail-Open

from audiosig import (
    time_stretch,
    AudioSignalError,
    InvalidParameterError,
    AudioShapeError,
)

def safe_time_stretch(audio, rate, fail_fast=True):
    """Time stretch with error handling."""
    try:
        return time_stretch(audio, rate=rate)
    except InvalidParameterError as e:
        if fail_fast:
            raise
        print(f"Warning: {e}")
        return audio  # Return original
    except AudioShapeError as e:
        if fail_fast:
            raise
        print(f"Warning: Invalid shape: {e}")
        return audio

# Fail-fast (default)
result = safe_time_stretch(audio, rate=-1.0, fail_fast=True)

# Fail-open (return original on error)
result = safe_time_stretch(audio, rate=-1.0, fail_fast=False)

Input Validation

import numpy as np
from audiosig import AudioShapeError

def validate_audio_input(audio, expected_sr=None):
    """Validate audio before processing."""
    if not isinstance(audio, np.ndarray):
        raise TypeError(f"Expected np.ndarray, got {type(audio)}")

    if audio.dtype not in (np.float32, np.float64):
        raise ValueError(f"Expected float32 or float64, got {audio.dtype}")

    if audio.ndim == 0:
        raise ValueError("Audio must have at least 1 dimension")

    if audio.size == 0:
        raise ValueError("Audio array is empty")

    return True

Performance Optimization

Memory Layout

import numpy as np

# C-contiguous arrays are faster
audio = np.ascontiguousarray(audio)

# Fortran-contiguous for column-major operations
# (not typically needed for audio)

Chunked Processing

import numpy as np
from audiosig import time_stretch

def process_large_file(audio, rate, chunk_seconds=10, sample_rate=24000):
    """Process large audio in chunks."""
    chunk_size = chunk_seconds * sample_rate
    results = []

    for i in range(0, len(audio), chunk_size):
        chunk = audio[i:i + chunk_size]
        processed = time_stretch(chunk, rate=rate)
        results.append(processed)

    return np.concatenate(results)

Type Selection

import numpy as np

# Float32: Good for most applications, lower memory
audio_32 = np.random.randn(24000).astype(np.float32)

# Float64: Higher precision, more memory
audio_64 = np.random.randn(24000).astype(np.float64)

# Processing preserves dtype
from audiosig import time_stretch
result_32 = time_stretch(audio_32, rate=1.1)  # Returns float32
result_64 = time_stretch(audio_64, rate=1.1)  # Returns float64

Integration with Other Libraries

With librosa

import librosa
from audiosig import resample

# Load with librosa
audio, sr = librosa.load("file.wav", sr=None)

# Resample with AudioSig (NumPy-only)
resampled = resample(audio, source_rate=sr, target_rate=16000)

With soundfile

import soundfile as sf
from audiosig import trim, peak_normalize

# Load audio
audio, sr = sf.read("file.wav")

# Process with AudioSig
trimmed, _ = trim(audio, top_db=40.0)
normalized = peak_normalize(trimmed, peak=0.95)

# Save
sf.write("processed.wav", normalized, sr)

With PyTorch

import torch
import numpy as np
from audiosig import time_stretch

# Convert to numpy
audio_torch = torch.randn(1, 24000)
audio_np = audio_torch.numpy().squeeze()

# Process with AudioSig
processed_np = time_stretch(audio_np, rate=1.1)

# Convert back to torch
processed_torch = torch.from_numpy(processed_np).unsqueeze(0)

Debugging and Profiling

Checking Intermediate Results

import numpy as np
from audiosig import frame_signal, frame_rms

def analyze_frames(audio, frame_length=2048, hop_length=512):
    """Debug frame analysis."""
    frames = frame_signal(audio, frame_length=frame_length, hop_length=hop_length)
    rms = frame_rms(audio, frame_length=frame_length, hop_length=hop_length)

    print(f"Audio length: {len(audio)}")
    print(f"Number of frames: {len(frames)}")
    print(f"Frame shape: {frames.shape}")
    print(f"RMS range: [{np.min(rms):.4f}, {np.max(rms):.4f}]")

    return frames, rms

Performance Profiling

import time
import numpy as np
from audiosig import time_stretch

def profile_time_stretch(audio, rate, iterations=100):
    """Profile time_stretch performance."""
    times = []

    for _ in range(iterations):
        start = time.perf_counter()
        result = time_stretch(audio, rate=rate)
        end = time.perf_counter()
        times.append(end - start)

    avg_time = np.mean(times)
    std_time = np.std(times)

    print(f"Average: {avg_time*1000:.2f} ms")
    print(f"Std dev: {std_time*1000:.2f} ms")
    print(f"Throughput: {len(audio) / avg_time / 1e6:.2f} Msamples/s")

    return avg_time, std_time