A real in-process Whisper engine — ggml parser, log-mel frontend, encoder, decoder, word-level timestamps — with native four-speaker diarization. No FFI. No Python. No subprocess. One binary that answers to agents first.
Most "Rust Whisper" projects wrap whisper.cpp over FFI or shell out to Python. franken_whisper implements the entire forward pass in safe Rust on FrankenTorch CPU kernels.
ggml model parser, log-mel spectrogram, and the exact BPE tokenizer — byte-for-byte compatible with whisper.cpp's model files. Drop a ggml-*.bin in place and transcribe.
Encoder and decoder with whisper.cpp's full timestamp-rule suite, greedy and beam decode, int8 quantization under a calibrated quality policy, and token-merging (ToMe) on the encoder — the lever behind the 2.99× row.
Cross-attention DTW word alignment — real per-word times from the model's own attention, not linear interpolation. Metal acceleration engages automatically on macOS.
Every row below ran the real whisper-cli incumbent side-by-side in the same
harness invocation, at matched greedy decode settings, with paired A/A null controls.
| Model / workload | Mode | Result | Transcript |
|---|---|---|---|
| large-v3-turbo, 124.5 s / 5 windows | whole job, no timestamps | 2.99× faster | WER 0.011 |
| tiny.en, 124.5 s / 5 windows | transcribe only | 1.52× faster | WER 0.0000 |
| tiny.en, 300 s / 10 windows | transcribe only | 1.51× faster | WER 0.0000 |
*Matched-greedy CPU comparisons (whisper-cli -bs 1 -bo 1), 32 threads, quiet x86-64 Threadripper host,
order-alternating pairs, per-engine A/A controls, CI95 gates. A result is admitted only when both same-invocation
A/A null medians sit in [0.98, 1.02] and the effect clears twice the widest null-CI edge. The full measurement
record, including every rejected optimization, lives in
docs/PERF_LEDGER.md.
A memory-safe Rust port of NVIDIA's Sortformer runs in-process by default, tracking up to four overlapping speakers. Turn evidence is fused into every segment, and speaker changes snap to the transcript's own punctuation.
Every segment carries its speaker, and speaker_segments gives you merged same-speaker runs with byte-faithful text and per-run confidence — no client-side join required.
Four lanes are a hard capacity boundary, not proof a recording has four speakers — the output says so explicitly (development_uncertified). Untimed audio stays unknown rather than guessed. Recordings past the boundary fall back to a bounded acoustic engine, and the report names which engine ran and why.
Robot mode streams sequenced, schema-versioned NDJSON. Errors are thirteen exact codes, not prose. Even argument-parsing failures come back as one parseable JSON object.
Dropped decode windows, ignored flags, and every non-fatal anomaly surface in
warnings and structured events — nothing important is stderr-only. Runs persist to
SQLite with replay envelopes; identical input and engine must produce an identical output hash.
Each stage has an independent time budget and emits paired start/ok events. Skipped stages say so — agents can tell "not needed" from "failed".
A 26,000-line negative-evidence ledger records every optimization that didn't work — with the measurement that killed it and the predicate under which it may be retried. Failed ideas are data, not embarrassments.
Performance levers must prove bit-identical output before their speed is even measured. The few accepted divergences live in DISCREPANCIES.md with kill switches and review dates.
Every run produces SHA-256 replay envelopes; a four-artifact replay pack reproduces any run byte-for-byte on another machine. Drift between engine versions is detected, never assumed away.
SHA-256-verified binaries for Linux (x86_64/arm64), macOS (Intel/Apple Silicon), and Windows. The installer also provisions both pinned model packages (~2.1 GB) so the default pipeline works offline.