The voice satellite

Work in progress

This is the newest thing in Nocturn and the least settled part of it. It works end to end and is covered by tests, but the wire messages, which actions a spoken session asks about, and the browser client — labelled a PoC harness in its own help text — are all still moving. Build against it only if you are willing to follow the changes.

Everything else in remote access is stable.

A satellite is a speaker in a room you talk to: a microphone, a speaker, and a ring of light. It says a wake word, you talk, it answers — and when what you asked for needs your permission, the ask still leaves the room for your phone.

The board

One board is supported: the Waveshare ESP32-S3-AUDIO-Board. Not “boards like this one” — the firmware in satellite/ is written against its microphone array, its codec and its LED ring, and nothing else has been built or measured.

That is a deliberately narrow claim rather than a modest one. Echo cancellation is the reason: what it leaves of the speaker’s own voice decides whether the board can hear a person while it is talking, and that depends on this microphone in this enclosure at this distance from this driver. A board with the same chip and a different acoustic layout is a different measurement, not a variant.

The ring

The ring is the whole interface, so it has to answer one question without being learned: may I talk now? Green means yes, and green is used for nothing else.

State	Ring	Meaning
boot	white, slow breath	Powering up
no-network	amber, slow breath	Not associated, or associated without an address
no-server	amber, spinning	On the network, looking for a server
fault	red, slow blink	Will not recover without a person
idle	dim blue, steady	Link up, waiting for the wake word
listening	green, breathing	A session is open — talk
thinking	blue, spinning	You stopped, nothing has come back yet
speaking	cyan, wave	The speaker is running
approval	magenta, blinking	Something is waiting for your decision — look at your phone
capture	red, steady	The microphone is being recorded for enrolment

Three choices in there are worth knowing, because each is a decision rather than a palette:

speaking is cyan, not green. While a reply plays the session is half duplex and the microphone is muted upstream, so talking over it reaches nobody. A green ring would be an invitation the board cannot honour.

Tempo carries meaning as much as colour. A slow breath is patience — boot, no network — and a normal breath is an invitation. The one flash in the whole design is green on connect, because connecting is a moment and not a state.

fault and capture are both red, and are told apart by motion. Fault blinks slowly; a recording is steady. Red-means-recording is understood by everyone who walks into a room, which is exactly why it is worth spending the colour twice rather than inventing a private one nobody knows.

The board decides all of this from what it observed, not from what the server believes. A round trip is 150–250 ms on a LAN and immediacy ends around 200, so anything a person perceives as slow — the ring lighting up when they speak — is local. Where the two disagree, the observation wins: the person is in the room and the server is not.

Configuring it

Spoken conversations need a second, different model: one that takes and returns a continuous audio stream rather than answering a turn at a time. It is not a setting on the chat model — it is a separate endpoint, and leaving it unset means a device asking for a spoken session is simply told this server has none.

GEMINI_API_KEY=your-key-here
GEMINI_LIVE_MODEL=gemini-2.5-flash-native-audio-latest

Variable	Meaning
GEMINI_API_KEY	the key for the live API
GEMINI_LIVE_MODEL	which live model — required, there is no default

Use that id, not a newer one

As of August 2026 gemini-2.5-flash-native-audio-latest is the only live model this works with — and, counter-intuitively, the newer ones are the problem. They do not support asynchronous function calling, and without it the whole conversation stalls the moment the assistant uses a tool: the model waits for the call to return before it will say anything.

That is unusable here rather than merely slow, because a tool call is exactly where an approval happens. “Let me check” would be followed by silence for as long as your phone takes to answer.

Reaching for the newest id is the natural instinct and the wrong one. If a later model gains asynchronous function calling, it becomes a candidate; until then this is not a version to bump.

There is no built-in default, deliberately. These ids change often, and one baked into the binary would either fail at connect time or — worse — connect and behave subtly wrong. Check what your own account actually offers rather than assuming: the list a key can reach and the list in anybody’s documentation are not the same.

One connection, one more domain

Voice is a domain on the same protocol as everything else, not a separate channel: voice.wake opens a spoken session, voice.credit says how much speech the device can hold, and the audio itself is binary frames — the only frames on the socket that are not commands. The server answers with voice.state on each transition and voice.interrupt when you talk over it, which tells the device to drop whatever it has buffered but not yet played.

One connection carries all of it because a second one would need a second copy of everything that already hangs off a device’s connection — pairing, the approval routing, presence. It also means a device that can already talk to the server can already speak, without another endpoint or another token.

Three ways this differs from chat, all deliberate:

• A device has one conversation at a time, which is why an audio frame carries no addressing of its own. The wake word already said which one.
• Audio is addressed, not broadcast. A conversation is shown on every device; speech is played on exactly one, or your phone would play what the speaker in the hallway is hearing.
• A spoken session ends when the device goes away. A chat turn deliberately survives that, because its answer is still worth reading when you come back. Audio has nobody to play to, and a live model bills for every second of it.

It also ends on its own, three ways, most polite first. The model hangs up — it is given a hang_up it can call when you say goodbye, answered by the driver and never part of the tool cage, because it steers the session rather than reaching into the world. Silence — about ten seconds after a reply with nobody speaking, which is long enough to ask a follow-up without the wake word and short enough not to leave an open microphone in the room. The budget — fifteen minutes, a cost guard rather than a conversation rule.

The silence window pauses while a tool is running or an approval is out on your phone. That is the one moment where hanging up would be worst: somebody has already been asked to do something.

Silence is judged by the device, not by what the model understood. Its detector fires about 200 ms after a person starts, on the echo-cancelled signal; a transcript arrives later and only for speech that made sense, which is the wrong question when you are asking whether anyone is still there.

Speech comes back credited, not paced. A live model emits a sentence far faster than it is spoken, so something has to decide the rate — and the only real clock in the system is the far speaker’s own crystal. The device grants its whole playout queue when a session opens and grants more as playback consumes it; the server sends what it has been granted and holds the rest. Nothing is metered by a timer, so there is no rate to get wrong, and overflow is structurally impossible rather than merely unlikely.

The window is deliberately much larger than the round trip that reopens it. Sized the other way — a buffer about as deep as the delay in refilling it — the queue is empty every time credit arrives, and no amount of tuning either number helps.

Between the server and one connection sits a short queue, four frames of it. A frame that finds it full waits rather than being discarded: the queue only fills when the socket behind it has stopped draining, which is precisely when nothing more should be sent. Dropping instead would put a gap in the middle of a sentence to solve a problem that waiting solves by itself. Speech goes to a device’s newest connection, since a device holds more than one while an old one times out.

Blocking the write until the device’s socket accepts it looks simpler and is not: a device blocks in its own receive callback, that callback runs on the task driving the connection’s keepalive, and a wait longer than the client’s read timeout looks to it like a dead socket. Flow control that drops the link under load is not flow control.

Interrupting a reply is currently off — the session is half duplex. While the speaker runs, the device sends silence upstream instead of the microphone, because the model must never hear the device’s own voice: what echo cancellation leaves of a reply still measures louder than a person, and no voice detector on either side can tell the two apart. Until the canceller is measured good enough, talking over a reply reaches nobody — let it finish. voice.interrupt remains the server’s message to drop buffered, unplayed speech, and voice.barge is reserved for the device-side detector (BARGE_IN_LOCAL, parked behind the same measurement).

A server with no live model configured refuses voice.wake and says so, rather than opening a session that can never answer.

Approvals in the middle of a sentence

A spoken session asks before it reaches the network, and the ask goes where every other ask goes: out of band, to a paired device. The room cannot approve what the room requested — answering “yes” by voice would be answered by the same room the instruction came from.

The wait is shown twice, because the two audiences are different. The model says it — the session is handed an interim result naming what is pending, so whoever is listening hears why the conversation stopped instead of hearing nothing. The device shows it — voice.state gains an approval state, so somebody who walks past a silent speaker can tell a pending decision from a broken session. Neither is allowed to fail the approval: showing a wait is worth less than the decision being waited for.

The state travels on the control queue, which overtakes audio, or it would arrive after the wait it describes was over. Clearing it says listening, because the device leaves that state by itself the moment speech starts playing — the clear only has to cover a refusal, where nothing follows.

Which actions ask is deliberately narrow: network reach, today, and nothing else. That is a measurement rather than a settled posture. Asking constantly is how people learn to approve reflexively, which costs more than it protects, and what is being tested is whether one asking kind earns the pause it puts in a conversation.

Knowing who is speaking

With NOCTURN_SPEAKER_MODEL pointing at a speaker-embedding checkpoint, the server recognises which enrolled voice is talking, continuously, from the audio a session is already sending. Enrolled voices live in the workspace’s voices.json; with none of them, or with no model, nothing below happens and the spoken path behaves exactly as it did.

Nothing waits for it. A session opens on the wake word as before; recognition runs beside the conversation and the answer is simply there by the time anything asks. That is not a shortcut — a live session’s system instruction travels in its setup frame and cannot be replaced afterwards, so identifying first would have meant pausing at the start of every conversation to learn something not needed until somebody asks something personal.

The result reaches the model two ways, and the second matters more:

• whoami returns {"name", "confidence"} when asked. An empty name means unrecognised, and the model is told not to guess which household member it might be. Outside a spoken session — a typed chat, an agent run — it always answers that, since nothing there is listening.
• On the request context, where a tool acting on somebody’s behalf reads it directly. A mail tool asked for “my messages” resolves the mailbox itself; the model never learns whose it was, cannot forget to pass it, and cannot be talked into naming someone else’s. The same shape as credential injection, where the guest never sees the value.

Spoken sessions now also carry the memory index, which they did not before: memory_read joined the voice cage. Reading only — memory_write stays out.

A recognised voice chooses context and address, never permission. Speech is a channel like the chat, where nobody authenticates the typist either, and a gated action is confirmed out of band on a second device in both.

Recording the uplink

Setting NOCTURN_VOICE_CAPTURE to a directory makes the server write what a device’s microphone sends to WAV files there — 16 kHz mono, the rate the link already carries.

It exists for one job: obtaining recordings of a voice through the microphone, room and echo canceller that will later have to recognise it, which a laptop microphone cannot stand in for. It is off unless the variable is set, because it writes what the room said to disk. Arming it logs a warning, every file is logged by name, the directory is created 0700 and the files 0600.

A recording is asked for rather than waited for:

nocturn enroll --device kitchen --seconds 60

This opens no conversation. capture.start puts the board in a state where the microphone streams and nothing else runs — no live model, no billing, no speaker, and no half-duplex gate muting the microphone whenever a reply plays. The ring goes steady red for as long as it records, which is a deliberate departure from the fault colour: that one blinks slowly, this one is steady, and someone walking into the room should be able to tell they are being recorded without knowing the palette.

capture.stop ends it, and so does the board on its own after two minutes. The second one is what matters: a caller that dies between start and stop must not leave a microphone streaming a room, and nothing in the house would say so.

Asking is a capability, not something any device may do. A paired app may (ClassApp), and the local command line may — it enrols itself once as ClassTool, which can start a recording and, pointedly, cannot approve a gated action. An appliance may not: a speaker in the hallway switching on the microphone in the bedroom is exactly the shape this refuses.

Recordings are cut by length and by silence rather than by session — a device opens a voice session on the wake word and never closes it, so waiting for one to end would mean never writing a file. A recording starts at the first sound, ends after four hundred milliseconds of quiet or eight seconds of speech, and is contiguous audio throughout: pauses inside it are kept, so what a file contains is what the microphone heard.

Turning recordings into a voice

Those files become a profile through the command line:

nocturn voices add --device kitchen lina /path/to/recordings/
nocturn voices ls
nocturn voices rm lina

add takes files or directories — a directory contributes its *.wav — and expects 16 kHz mono 16-bit WAV, which is what the uplink already writes, so nothing captured needs converting. It reads each recording, embeds it, and appends the result to the workspace’s voices.json. Embedding needs NOCTURN_SPEAKER_MODEL; without it the command says so and does nothing.

--device is required and never guessed. A voice enrolled on a phone and the same voice heard through a hallway speaker are two channels, not one — different microphone, different room, different echo canceller — and the take records which one it came through. Enrolling somebody a second time on a second device therefore adds that channel rather than replacing the first: a person’s score is their best single take, so having enrolled twice can only help them.

The command opens no workspace and talks to no server. It edits one file, and a server started afterwards picks it up. An unreadable recording is skipped with its name on stderr and counted in the summary, because a silently thinner profile is one that recognises worse for a reason nobody sees.