Parakeet vs whisper.cpp on a laptop CPU: 6.4 s to 0.7 s per take

Push-to-talk dictation is unusable if it thinks for five seconds after you let go of the key. Ours did. Getting it under a second took three changes, and the one that mattered most was the one we would have bet against: using fewer CPU threads.

what was measured

An i5-10300H laptop - 4 physical cores, 8 logical - running Windows. No GPU involvement: everything below is CPU. The engine is sherpa-onnx with NVIDIA’s Parakeet TDT 0.6B v3, int8 quantised, against whisper.cpp with large-v3-turbo q5_0 for comparison. Times are wall clock from the end of speech to text in hand, medians of repeated runs on the same audio.

Two clips recur below: a short one, 3.9 seconds, which is what a dictation take actually looks like, and a long one, about 24 seconds, where decoding dominates.

where the 6.4 seconds went

The first implementation ran the sherpa-onnx command line tool once per take. For the 3.9 second clip that took 6,100-6,500 ms, of which 4,400-4,800 ms was loading the model. We were paying for the model load on every single press of the hotkey.

This is the whole game on CPU inference and it is easy to miss, because a benchmark of your decoder looks fine in isolation. The fix is to stop exiting: sherpa-onnx ships sherpa-onnx-offline-websocket-server, which keeps the recognizer in memory and takes audio over a socket. We keep one alive per model and thread count, on loopback, and shut it down after 15 minutes idle so a 600 MB resident model does not sit there all day.

the thread count, which we got backwards

The one-shot tool had been running with logical cores minus one - 7 threads. The obvious move when latency matters is to give it everything. Instead:

Twice as slow with twice the threads. Hyperthreads share the execution units that the matrix multiplication in onnxruntime already saturates, so the extra threads add scheduling and synchronisation and take back nothing. We had been throwing away half our speed by asking for more.

Which means std::thread::available_parallelism is the wrong function - it counts logical processors. Count physical cores: GetLogicalProcessorInformationEx with RelationProcessorCore on Windows, and on Apple Silicon hw.perflevel0.physicalcpu, which gives you the performance cores rather than the total including efficiency cores.

two more levers on the same path

Warm up while they are talking. Even resident, the first request after launch pays the load. Trigger it the moment recording starts rather than when it stops: the person speaks for a second or two anyway, and the load hides inside that.

Trim silence, but only for live takes. A push-to-talk recording carries dead air at both ends from human reaction time, and the decoder charges for it. Cutting it is free speed. Do not do it to a file the user is transcribing for subtitles - its timings become the subtitle timings, and you have just shifted every cue.

the result

About nine times faster, and none of it came from a better model.

against whisper.cpp

On the ~24 second clip, same machine:

Both were accurate, including on Polish, which we did not expect from a 0.6B model. The comparison is not apples to apples - large-v3-turbo is far bigger - but it is the choice you actually face when picking a default for a laptop.

Whisper stays in the product anyway, because Parakeet gives us no timestamps, no translation and no prompt biasing. Subtitles need word timings. Translation needs a model that translates. And a vocabulary of names and jargon gets into Whisper through its initial prompt, which Parakeet has no equivalent for. So dictation runs on Parakeet where it is installed, and anything involving a file or another language goes to Whisper regardless of what is selected.

what we did not get

Sub-300 ms is the number people quote for models in this family, and we are not at it. At a real-time factor around 0.17, the arithmetic says you cannot be: a 4 second take costs ~700 ms of decode no matter how warm the model is, because the decode only begins when the speech ends.

The only way past that is to stop waiting - run a voice activity detector over the incoming audio, ship each segment as it closes, and leave just the tail to decode when the key comes up. That is a different architecture, not a tuning pass, and it is the honest answer to “why is it not 300 ms”: because we have not rewritten it to decode during speech yet.

if you are doing this yourself

• Measure the model load separately from the decode. It is probably most of your time.
• Try physical-core thread counts before anything else. It is one line.
• Start loading on record-start, not on record-stop. Users measure latency from when they stop talking.
• Put the number in the interface. Ours prints the time under the test transcript, so a regression is visible to somebody other than the person waiting.

Who wrote this. We build owntools. These are our own measurements on one laptop, which is a sample of one - if your numbers differ, particularly the thread finding on a machine with more cores, we would genuinely like to hear it.