This is Interpretab, a Chrome extension. It listens to whatever a tab is playing and interprets
it out loud. It also works the other way round: your own voice, translated, and sent to the
person you are talking to. No third-party server sits in the middle. The connection and the API
key both stay inside the browser. Let me show you the demo first.
Demo: translating a tab
Any tab
A video, a webinar, a podcast. The source language is detected, not declared.
Spoken
The translation out loud, with the original ducked to 15% underneath it.
Subtitled
Three rolling lines on the page itself, plus a transcript in the side panel.
Interpretab
The clip runs itself. Point at the subtitle line first, then at the transcript in the
panel, then at the language picker when it switches to Chinese.
Demo: translating your voice
Your voice
Picked up as you speak, and interpreted while the sentence is still going.
Two-way
What you said and what they said, both translated, side by side in one panel.
Into the call
Meet lists
Interpretab as a microphone. Pick it and the far end hears the translation.
Interpretab
Second half is the two-way mode: one panel, two languages, and it works out which
direction each sentence is going.
Interpretab architecture
Interpretab
Okay, the shape of it. Two main components. Web Audio is the browser's built-in audio
processing; it does the capture and the resampling.The Gemini Live API takes audio in and
streams the model's answer back over a WebSocket, in real time.The whole application lives in
one offscreen document, a page you never see.Three audio contexts, one per sample rate.And
two WebSockets, one for the tab audio and one for the microphone.
The two Live models
Interpretab
Interpretab uses two Live API models. The simultaneous one starts translating while you are
still speaking, and takes no instruction up front.The conversation one waits for the speaker
to finish. That makes it slower, but the quality is higher, it costs less, and it is the one
that can take a glossary.
How it integrates with Meet
No extension can register an audio input with the system. One page is another matter.
Interpretab
Next, hooking this into Meet to interpret both ways. A Chrome extension cannot register its
audio output as a microphone with the operating system. Inside the Meet page it can.We add a
script to the world where Meet's own scripts run, and it puts a microphone called Interpretab
in the device list.The user picks it in Meet's settings, and the translated audio reaches the
far end.One catch: even inside the same page, Meet's side and the extension's side run in
separate worlds, and a device list changed from the extension's side is not the one Meet sees.
So there is one script on each side, bridged with postMessage.Measured on a real call, the
end-to-end delay is about three seconds.
The hard parts
Three things that fought back
A session lasts about ten minutes
A connection drops and you cannot tell why
Finding a new model, and moving to it
Interpretab
Three problems that came up while building this. I will go through them one at a time.
Session lifecycle
A Live API socket is capped at about ten minutes. That surprises most people. The server
warns you at nine, gives you fifty seconds, and means the deadline literally.
Interpretab
A Live API WebSocket closes after about ten minutes. Past nine minutes the server sends goAway,
and it cuts fifty seconds later.So Interpretab opens a new Live API session the moment goAway
arrives, and swaps at whichever comes first: the turn finishing, five seconds of quiet, or one
second before the deadline.Whatever the old session never finished translating is sent again
on the new one.The API has its own sessionResumption. We tried it and do not use it: the
model reads its last sentence out again.
Error handling
When a WebSocket closes, all the browser gets is a close code. No reason comes with it.
A bad key, a spent quota, a network fault. All the same empty close, with
nothing in it to tell them apart.
So ask over HTTP first
One GET /v1beta/models with the same key, before the socket opens.
400 means the key was rejected. 401 means the key is the wrong shape.
close 1011 · cut off mid-run
Cross the quota during a run and a reason does come back. But a close frame
holds 123 bytes, so the tail is gone. Match on the start of it.
Both behaviours were watched on real API calls. For the quota one,
tests/quota-close.mjs drops the free-tier ceiling to a hundred tokens to reproduce it.
A spent quota still lets the socket open, so ten retries all connect and all die. It comes back
at midnight Pacific, which is a good reason to stop retrying.
Interpretab
When a WebSocket closes for whatever reason, all the browser gets is a number, a close code,
with no reason attached. A wrong key, not enough quota, a network problem: the same close code
for all of them.So before the socket opens we ask over HTTP whether the key works, and read
the status code.Cross the quota in the middle of a session and a reason does arrive, but it
is cut off at a hundred and twenty-three bytes.Neither one is worth retrying.
Automatic migration to new Live models
Interpretab
Live API models turn over quickly. Hard-code a model name into the build and one day it stops
working.So once an hour a GitHub Action calls each model we use, for real. Is it still in the
list, and does a session still open with the production setup?When a retirement is detected,
and otherwise once a day, it goes looking for a new model. Only the ones that actually opened a
session stay in the running.That gets committed to the config file,every client reads it at
startup and every six hours,and the switch happens at the next socket handover, in the middle
of a run. No store re-review.
Long-running end-to-end soak testing
Soaks against the real API
tests/soak.mjs speaks synthetic sentences through a live
session for as long as you leave it, and has a second model grade every translation.
All three use cases behaved the same. Six
goAways in the hour, a handover each time, seven sessions chained end to end, and not one
closed by the server. Even the five that handed over mid-sentence still passed grading.
Use case
Length
Sentences
Passed
Turn complete
Mic, conversation model, en → ja
60 min
281
99.3%
1.60 s
Mic, simultaneous, en → ja
60 min
198
92.4%
0.98 s
Tab audio, en → ja
15 min
51
92.2%
0.23 s
An hour of ja → en on the same tab-side code scores well below these, because
the input was flat synthetic Japanese. Every report is in the repository.
Interpretab
We ran end-to-end soaks against the real application.Three use cases, sentences sent one at a
time, and a second model grading the transcription the Live API returns, one sentence at a
time.Six goAways in the hour, a handover each time, seven sessions chained end to end, and
nothing closed by the server. Even the ones that handed over mid-sentence passed grading.
Lessons learned
01
Call the real API first. Two of the three protocol surprises we hit
here were in no document anywhere.
02
A WebSocket just closes and never says why. Ask the same thing over
HTTP and a status code tells you.
03
The Live API hears the end of a turn in the audio it receives (VAD).
Keep sending silence while the speaker is quiet.
04
Do not bake anything that changes into the build. Model names and
prices move on Google's schedule, so read them from a file.
Retry a flaky network, and nothing else. A spent quota or a bad key gives you the same
answer however many seconds you wait.
Interpretab
Four things I learned building this. First, call the real API. What it actually does when
something unexpected happens is usually in no document anywhere.Second, when a WebSocket
alone will not tell you, ask the same thing over HTTP.Third, how the Live API detects the end
of a turn. Do not stop sending audio when the speaker goes quiet. Keep sending silence, or the
model waits forever.Last, do not bake anything that changes into the build. Model names
and prices move often.
A personal project. Not made, supported or
endorsed by Google. It is machine translation, so it mishears, and it sometimes says something
the speaker did not.
Interpretab
It is on the Web Store,the guide is on GitHub Pages,and the code is Apache two point
oh.Personal project, not a Google product. Thanks very much.