Whisper hallucinates
on silence.
Speech recognition does not go quiet when nobody is talking — it invents, confidently. Gating on voice activity is the fix, and the standard way to set that gate silently throws away thirty seconds of somebody presenting.
Ampersand can record a meeting — in a room, or a call where it captures the far end too — and turn it into a note with a summary, the decisions, the action items and the full transcript. Speech recognition happens on your own machine, with a Whisper model running through whisper.cpp on the GPU via Metal. No bot joins the call, and no audio is uploaded anywhere.
Cloud transcription was an explicit non-goal rather than something we ran out of time for: an hour of a private conversation, including whoever else was in the room, is the most sensitive thing this app would ever transmit. Running the model locally is the easy part. Deciding what to hand it is where the work was.
- the speech model, on your Mac
- ~570 MBthe speech model, on your Maclarge-v3-turbo, q5-quantized — fetched once, on first use
- the faster, rougher option
- ~140 MBthe faster, rougher optionbase — for when the big one is more than you want to store
- languages you can pin
- 43languages you can pinplus Automatic; a set language is what makes a wrong guess impossible
A meeting is mostly silence
Speech recognition does not return an empty string when nobody is speaking. Whisper was trained on a large corpus of transcribed video, so its idea of “audio with no speech in it” is shaped by intros, outros, background music and subtitle credits. Feed it room tone and it will confidently produce something.
# the shape of it. the model was handed every second
# of the meeting, including the ones nobody spoke in.
00:14:02 Thank you.
00:14:09 Thank you.
00:14:16 Thank you.
00:14:23 Subtitles by the Amara.org community
00:14:31 Thank you.
# nothing failed. nothing warned. the model was asked
# what was said, and it answered.
The repeated “Thank you.” and the fabricated subtitle credit are the two that everyone who has pointed Whisper at a long recording has seen. On a podcast they are a curiosity; in a transcript somebody will read as a record of what was said they are a defect, and nothing in the output distinguishes them from the real lines.
Now the ratio. A meeting has pauses between sentences, silence while somebody finds a document, the four minutes at the start where people are still joining, and a stretch after the goodbyes. Measured in seconds, a meeting is mostly silence — so gating on voice activity is not a performance optimisation that saves GPU time. It is a correctness requirement.
The textbook threshold throws away speech
The standard energy-based gate is two lines. Split the audio into short frames, take the energy of each, and estimate the noise floor as a low percentile of those energies — the quietest tenth of a recording is a fair guess at the room with nobody in it. Then call anything a fixed multiple above the floor speech, with an absolute minimum so a digitally silent source does not produce a threshold of zero.
# the textbook gate: a noise floor taken from a low
# percentile of frame energy, times a multiplier.
floor = percentile(frame_energy, low)
threshold = max(absolute_floor, floor * voice_multiplier)
# thirty unbroken seconds of one person presenting:
# the low percentile lands INSIDE the speech, the
# multiplier lifts the threshold above every frame in
# the segment, and the whole stretch is classified as
# silence and dropped. no error. no output.
# the fix is a third term -- a ceiling on the threshold,
# taken from a high percentile of the same signal.
speech_peak = percentile(frame_energy, high)
threshold = min(threshold, speech_peak * peak_fraction)
The floor estimate has landed inside the speech, so the threshold derived from it clears the loudest frame in the segment. Nothing is above the line: thirty seconds of somebody presenting is classified as silence and dropped.
The same frames, the same floor estimate. A cap taken from the peak holds the threshold under speech that is plainly speech, whatever the floor says. Every frame is kept.
The estimator assumes the segment it is looking at contains some silence. Take one that does not — thirty unbroken seconds of a person presenting — and the low percentile no longer lands in the room tone. It lands in the quieter part of the speech. Multiply that by the voice factor and the threshold sits above every frame in the segment: the gate looks at thirty seconds of continuous talking and reports that nobody said anything.
The repair is a third term — cap the threshold at a fraction of a high percentile of the same signal. If the loudest part of this segment is plainly speech, the threshold may not exceed a set share of it, whatever the floor estimate says. One extra line, and the all-speech case stops being a hole.
It was found by a test, not by review. The gate had been read by a human and looked correct, because it is the correct formula for the case it was written for. What caught it was feeding it audio that was entirely speech and asserting that the boundaries came back non-empty.
And it is the more dangerous of the two directions. A hallucinated sentence is visible: it is in the transcript, it reads oddly, somebody notices. Audio that was silently dropped looks exactly like a pause. Nobody reads a transcript and thinks “there should be thirty more seconds here”. Of the two ways this gate can be wrong, one reports itself and the other does not.
Where the cuts go, and why overlap is not free
Who was speaking: a fact on a call, a guess in a room
On a call we capture two audio tracks: your microphone, and the audio your Mac is playing. Those are separate signals from separate sources, so labelling one you and the other the far end is not inference and cannot be wrong. The two-track split is the diarization — no model, no embeddings, no chance of a confident mistake.
In a room there is one microphone and four people in front of it. Real multi-speaker diarization means computing voice embeddings and clustering them, and it is wrong often enough that a transcript would sometimes attribute a sentence to the person who did not say it. Putting the wrong quote in a colleague’s mouth is worse than attributing nothing, and shipping that in a first version is how a feature earns a reputation it never loses.
So an in-person transcript carries no speaker labels at all, and the note says so in a quoted line closing the transcript. Stating a limit inside the artifact costs one sentence and stops somebody discovering it three months later while relying on it.
Two signals from two sources. No model is involved, so there is no confident mistake available to make.
One signal, several people. Telling them apart needs voice embeddings and clustering — a guess, and not one we are willing to put in somebody’s mouth.
A wrong language does not fail. It translates.
The bug that took longest to see had nothing to do with audio. Whisper can detect the spoken language, and if you do not tell it one it guesses per decoding window. A wrong guess does not error and does not warn — it translates. A Serbian conversation came back as fluent, plausible English.
That is the same failure class as the silence hallucination: confident, well-formed output, wrong in a way the output itself cannot express. We had detection. It never ran on the path most people use.
# what the loop used to do
for segment in session:
if already_transcribed(segment):
continue # <-- detection lived below this
if language is unknown:
language = detect(segment)
transcribe(segment, language)
# a session that is COMPLETE and has no language never
# reaches the detector. that is every recording started
# from inside a note, because those transcribe while you
# are still talking.
Detection sat inside the per-segment loop, below the early exit for segments already done, so it could only ever fire on a segment that also needed transcribing. Every combination worked except one: a session that is complete and has no language recorded, which is precisely what a recording started from inside a note produces, because those transcribe while you are still speaking. By the final pass there is nothing left to transcribe, so nothing to hang detection off, and decoding fell back to the per-window guess. The path that worked in testing was the one nobody uses.
The repair had three parts, and only one of them is the bug fix.
- The three-way decision — skip, detect only, or transcribe — became a small named function with its own tests, precisely because the conditional was the thing that was wrong. A mutation of it now fails a test instead of a meeting.
- Detection runs once per session, on the longest available window. Whisper reads only the opening of what it is given, so a one-second window is close to a coin flip and a half-minute one is not. The old code sampled the first segment — the worst possible one: “can you hear me?”, room tone, somebody still joining.
- The verdict is pinned to the session and persisted immediately, so the repair case — which reaches the loop with nothing left to transcribe — does not recompute it forever.
We also log which of three routes decided the language — set explicitly, carried over, or detected — with the confidence and the window length, because a run that never detected anything used to be indistinguishable from one that never tried. And the override lists 43 languages: if you work in a language other than English, setting it is what makes a wrong guess impossible rather than merely recoverable.
Where the audio goes, and the one thing that can leave
A two-hour recording cannot sit in memory, and a crash ninety minutes in must not cost you the meeting, so audio is buffered to a folder outside your vault — never into iCloud or whatever else syncs it. On a Mac that can store a key for it, that buffer is encrypted at rest with a key belonging to that machine alone, not with your vault passphrase. Recording never asks whether your vault is unlocked, a decision we got wrong the first time and wrote up separately.
It becomes eligible for deletion only once its content exists somewhere that is not the buffer, so a crash between stopping and writing cannot cost you the recording.
Transcription itself is entirely local. The one thing that can leave the machine is the finished transcript, if you have pointed AI summarization at an off-device provider — a reasonable thing to do, which the app discloses permanently in its settings, naming the provider it would go to. That line appears only when the provider really is off-device: a local model or Apple Intelligence sends nothing anywhere and gets no warning, because a warning on the route that carries no risk is how people learn to ignore the one that does. It replaced a per-recording confirmation dialog, which was right about the substance and wrong about the placement: the provider is chosen once, so re-asking every time was friction with no decision in it.
What we took from it
- A model that cannot say “nothing” will say something. Wherever you hand a generative model an input that might be empty, decide what empty means before it does.
- Estimators carry assumptions about their input. A noise floor from a low percentile assumes there is noise to find. Test the distribution you will really see, degenerate ends included.
- Prefer the failure that announces itself. Between output that is visibly wrong and output that is invisibly missing, test the second — nobody reports a gap they cannot see.
- Do not promise inference you cannot stand behind. Two microphones are a fact; one microphone and a clustering model is a guess, and an unlabelled transcript beats a confidently mislabelled one.
The meetings overview covers what the feature does day to day, the documentation every setting behind it, and the technical specifications the rest of the app.
Recorded on your Mac.
Transcribed on your Mac.
Ampersand records a meeting, transcribes it on-device, and writes the note — without a bot joining the call or a byte of audio leaving the machine.
Free to start · Mac, Windows & Linux · No account required