From Pager to Walkie-Talkie: My M5Stack Now Converses with Mistral Vibe

A direct sequel to "Mistral Vibe Without the Tether: Why I Built a Hardware Pager". Back in May, the M5Stack could alert and validate. Three releases later (0.4.0 → 0.6.0), it configures itself, listens, and talks. The story of what changed — on the device side, the Mistral Vibe side, and the working-with-an-agent side.

TL;DR #

• The tethering pager became an agent walkie-talkie: I dictate an instruction into the M5Stack's microphone (long-press A), Voxtral transcribes it, the turn starts in Vibe — and the final answer is read out loud through the Fire's speaker (Voxtral TTS). The conversational loop is closed, keyboard untouched.
• May's plea was answered. Mistral Vibe 2.23 replaced AgentLoop.set_approval_callback with an InteractionRequestBroker: approvals are now events carrying a request_id, resolved through a public API. The "TUI modal vs physical button" race became native — and I deleted the entire monkey-patch.
• v0.4.0: an on-device config menu (quiet mode, LED brightness, mic source, voice language…) persisted in NVS — plus a "model" selector offering the Fat Kitten 😼, "the new Mistral model".
• The hidden cost of on-device TTS: a 512-byte Bluetooth SPP queue that dropped bytes mid-stream, an I2S DAC playing 5.5× too fast (the "sped-up voice"), a Windows asyncio.sleep that chopped the bitrate. Three bugs, three embedded lessons.
• The making-of is a story of its own: fifteen briefs written for Mistral Vibe (the implementer), one rule that changed everything — "compile before every commit, or it's a failure" — and one git reset --hard too many, recovered through git's dangling blobs.

1. Where we left off #

At the end of May, vibe-m5stack did one thing very well: breaking the tether. When the agent asked permission to write a file or push a commit, the M5Stack Fire lit up, the LEDs were visible from five meters away, and you validated with a button — from the kitchen. The transport had moved from USB to Bluetooth Classic SPP, the firmware displayed a continuous ambient status (thinking, reading, writing, waiting for me, dead), and a watchdog alerted when the agent crashed without a word.

But the device remained a receiver. It showed me what the agent was doing, it rang when it needed me — and there was no way to answer back without returning to the keyboard. May's roadmap listed four possible follow-ups; the fourth — "voice button to give complementary instructions without coming back to the PC" — was the most ambitious.

One summer and four releases later (0.4.0, 0.5.0, 0.5.1, 0.6.0), it shipped — along with things I hadn't dared put on the list. Here's the detail, release by release, with the real stories.

2. v0.4.0 — The device gets its settings #

First brick, less spectacular but foundational: a configuration menu directly on the device. Long-press C (~1 s) from the home screen, navigate with C (next) and B (previous), select with A. Every setting is persisted in NVS (the ESP32's flash memory) with a validation magic byte and safe defaults on a fresh device:

• Quiet Mode — mutes approval vibrations and beeps (for meetings);
• LED Brightness — 16 / 32 / 64 / 128 / 255, applied live;
• Model — the animated Mistral cat… or the Fat Kitten 😼;
• Mic — push-to-talk source: device or PC;
• Voice Out and Voice Lang — arrived later, more on that below;
• Debug and Demo Mode — flow auditing and standalone showcase.

The Fat Kitten deserves an explanation. It's an easter egg: a fixed pixel-art sprite, white body with black outline, too fat to dance — that's precisely the joke. The screen displays a fake banner reading "the new Mistral model". You pick your model like a wallpaper, and the obese cat stays impassive while the real agent works.

The embedded anecdote: edges vs levels #

The first version of the menu had a delicious flaw: it closed by itself within 30 ms. The cause: the button edge (the exact instant of the press) opened the menu and navigated, but the level "button still held" — read a few loop iterations later — triggered the exit. In embedded electronics, the edge/level distinction is the first thing you learn, and the first thing you'll forget in every new context. The fix: purge flags when the menu starts, wait for release (wait-release) before entering the navigation loop, and base ButtonManager::isHeld() on a real level rather than a residual edge.

It's the kind of bug that shows no mercy and only appears on real hardware. Two lines of fix, one hour of debugging.

3. v0.5.0 — Talking to your agent #

This is the release that changes the nature of the object. The principle: long-press A (~0.5 s), speak, release. The transcribed text appears in Vibe as a real message and starts a turn. Three gestures, depending on context:

The second gesture is the one that changes the daily routine: the agent proposes a commit, I say "A long… approve, but add the tests before pushing" — the approval fires on button release and the comment drives the todo list live, without blocking the turn. The third gesture's asymmetry is intentional: rejecting takes longer than approving, because a reason-less rejection forces the agent to guess.

How it works (and what it cost) #

The Fire has no "official" microphone on its spec sheet, but its M5GO base carries a perfectly usable MEMS. The outbound voice pipeline fits in four steps:

M5Stack Fire (MEMS mic on the M5GO base)
   │  ADC capture → G.711 µ-law 16 kHz (anti-aliasing decimation)
   │  streamed LIVE while dictating (base64 chunks, Bluetooth SPP)
   ▼
PC (vibe-m5stack plugin)
   │  reception + resampling (ADC rate auto-calibration:
   │  the device measures its own recording duration)
   ▼
Mistral Voxtral (transcription)
   │  API key resolved exactly the way Vibe resolves it
   ▼
Vibe (text injection → agent turn)

Two architectural choices deserve explanation.

Audio streams while you speak, not after. On button release, transcription starts immediately — perceived latency is Voxtral's, not the transfer's. The price: the Bluetooth SPP link now carries a real-time stream on top of the JSON status messages, which would become the heart of the v0.6.0 problem.

The API key isn't reconfigured. The plugin resolves the Mistral key exactly the way Vibe does (environment variable, or the browser-login keyring). Zero extra configuration: if Vibe works, voice works. It's the kind of detail that separates a weekend project from a tool you keep.

Up to 60 s of dictation, LISTENING / TRANSCRIBING states on screen with their dedicated LED animations, and a PC-mic fallback if the device lacks an M5GO base.

Demo mode: the object without the PC #

A bonus that appeared while voice-dictating — from the couch — the feature's own specification: Demo Mode. With no PC session, the device chains its seven LED animations ten seconds after boot (welcome, the four thinking activities, waiting, done), each state captioned on screen in its color, animated cat or Fat Kitten in the spotlight. Any button exits. It's the showcase for a conference booth — and proof the device is self-sufficient.

And one last discreet piece: hot reconnection. Rebooting the device (flash, Bluetooth dropout) no longer kills the session — automatic reconnection and resynchronization. Before, every reflash was a lost session; now it's a non-event.

4. v0.5.1 — The day an upgrade killed the device (and why that's good news) #

This is the section I've wanted to write since May.

In the original article, the AgentLoop.set_approval_callback monkey-patch was presented as an accepted trade-off: "the wrapper's robustness depends on Vibe's internal API. If Mistral refactors AgentLoop, the wrapper breaks." And I made a wish: "an official Mistral extension point for the permission layer."

Mistral delivered it. And like any internal API migration, it started with a crash.

The crash #

uv tool upgrade one August morning → Vibe 2.23 → AttributeError: set_approval_callback at launch. Version 2.23 removed the method I was patching, replaced by an InteractionRequestBroker: approval requests become ApprovalRequestEvents emitted in the agent loop's event stream, each carrying a request_id, resolved through a public API — resolve_approval_request(request_id, response, feedback).

Immediate consequence: nobody could install the plugin anymore (the resolver pulled an incompatible Vibe). I set an emergency pin (mistral-vibe<2.23 — v0.3.1 had already served as degraded mode), then ported the hook.

The migration: less code, more guarantees #

The new mechanics are objectively better, and not just because they're official:

• The hook observes the event stream it was already iterating, spots ApprovalRequestEvents, and launches the M5Stack race as a background task — without consuming the event, which the TUI still receives.
• Device and TUI resolve the same request_id; the broker ignores late resolutions (future.done()). The "first to respond wins" race — which I used to simulate by hand with a shared Future — became native.
• No more digging through the Textual TUI's internals to close the modal when the button won: that hack died too.
• An important semantic detail: on the legacy path, a device timeout returned "NO" — acceptable when racing our own modal, dangerous on the new one (a phantom rejection would steal the decision from the TUI). Now, timeout = we don't resolve; the decision stays with the TUI.

All the legacy code is gone: patch_agent_loop(), the modal wrapper, the _pending_approval hack. One path remains, plus a readable fail-fast at import — if an old API is detected, the message says exactly what to do (uv tool upgrade mistral-vibe) instead of a cryptic AttributeError. Pin lifted; 0.5.1 requires mistral-vibe>=2.23.

The git anecdote: staged objects never die #

During this migration, the implementing AI ran an unsolicited git reset --hard — apparently wiping part of the session's work. Except git throws away almost nothing: staged blobs stay in the object store until the garbage collector runs. A git fsck --lost-found, spotting orphaned blobs by date and size, a git cat-file -p to verify content — and everything came back.

A double lesson: for me, dangling blobs are a real safety net, not a blog legend; for the agent workflow, it's the definitive argument for the rule that now governs the whole project — "no git reset --hard" is written at the top of every brief.

5. v0.6.0 — The agent answers out loud #

With outbound voice, the object was half a walkie-talkie: I spoke, the agent executed — but its answer stayed trapped in the TUI, ten meters away. Version 0.6.0 closes the loop: at the end of every turn, the agent's final answer is read out loud through the Fire's speaker.

The setting fits in two menu items:

• Voice Out: Off (default — strict opt-in) / Device (the Fire's speaker) / PC (the workstation's speakers);
• Voice Lang: FR (Marie voice) / EN (Jane voice) — applied to the next playback, no restart.

Synthesis reuses the TTS client embedded in Vibe itself (Voxtral mini-TTS model) — same key, same resolution, same philosophy as dictation. On the plugin side, the turn's text is cleaned before playback: code blocks and markdown are stripped (nobody wants to hear "triple backtick"), a technical guardrail caps it at 4,000 characters, and playback is complete — any button interrupts it, a new dictation cuts it too.

The outbound audio path: PC-side synthesis → re-encoding to G.711 µ-law 16 kHz (the link's format, symmetric to outbound voice) → ~90 % peak normalization → anti-aliasing low-pass → 24→16 kHz decimation → base64 chunks streamed over SPP → 1.5 s device-side pre-buffer → playback. A tts_diag telemetry reports chunks/bytes received back to the PC at the end of every stream.

Sounds simple. It wasn't.

Bug #1 — The 512-byte Bluetooth queue #

First real test: the voice played for a few seconds, then chopped itself up. The firmware log spewed RX Full! Discarding in a loop. Diagnosis: the SPP receive callback had a 512-byte queue — plenty for JSON status messages, derisory against a continuous audio stream. Every byte arriving while the queue was full was silently dropped.

Fix: a dedicated task (core 1, 1 ms cycle) continuously drains the queue into a 16 KB ring buffer, and the PC cuts its chunks to ≤ 390 bytes to never overflow it. The general lesson: a serial receive API is always designed for messages, never for streams — the moment you carry audio, reception becomes an architecture problem, not a callback.

Bug #2 — The DAC that played 5.5× too fast #

Next, the voice played — but sped up, fast-forward cassette style. Cause: the ESP32's internal DAC clock (GPIO 25, the Fire's 1 W speaker). Configured for the target rate, it behaved chaotically below a 4 kHz request — its real minimum being ~22 kHz. Result: the stream played 5.5× too fast.

The DAC won't go lower; it's a hardware constraint. The workaround: compensate with sample duplication (zero-order hold) — measure the actual ratio at every playback and upsample the stream so the final duration is right. It's a sparrow-level hack, it's measured, and it sounds right.

Bug #3 — The asyncio.sleep that lies #

Last bug, the sneakiest, because it was invisible in the code. The plugin paces chunk sending to hold the bitrate; the first version slept 15 ms between sends. On Windows, asyncio.sleep(15 ms) actually sleeps ~21 ms — and that drift is enough to drain the device-side buffer between two chunks. Choppy voice, no firmware-side explanation, because the firmware was innocent.

Fix: switch to absolute deadlines (next send at t0 + n × interval, regardless of actual sleep time) and lock the bitrate at exactly 16 KB/s. Lesson: with real-time software, never reason in sleep durations — always in deadlines.

And the small ones #

Three lower-visibility fixes complete the set: residual presses (a button edge older than the playback killed it on the first frame — purge at playback start, PC→device cancellation notification, 12 s no-data watchdog); multi-message RX (one JSON line per loop iteration couldn't keep up with a TTS stream's ~60 lines/s — up to 12 messages processed per iteration); and I2S sharing between mic and speaker (strict mutual exclusion: no playback during capture, a dictation first stops playback then releases the driver).

6. The making-of: fifteen briefs and one rule #

One detail deserves telling, because it says something about how we build with agents in 2026: this project wasn't written by hand.

The workflow has two roles. Mistral Vibe is the implementer: every feature starts from a written brief (fifteen BRIEF_*.md files at the repo root) that locks in decisions up front — architecture, protocol, acceptance criteria, and rule 0: "pio run + pytest before every commit. No git reset --hard." And I hold the other role: decision-maker and reviewer. The trade-offs (opt-in by default, full playback, approve/reject asymmetry) are human, written into the brief before a line exists — and the brief doubles as the review grid: produced code gets read against it, criterion by criterion, and anything that drifts from the contract shows immediately.

What worked remarkably well: everything structured and isolated — the ConfigManager with its NVS magic byte, the separated audio modules, the hardware-free pytest suite. What broke twice: state machines and cross-cutting integrations — the first menu version (unusable, see the edge/level anecdote), and two failed passes on voice, recovered by reframing the brief rather than patching the produced code.

The rule that most changed the yield isn't technical: "compile before every commit, or it's a failure." Without it, the agent stacked broken commits and debugging drowned in the diff. With it, every branch point is a runnable state — and rollback stays a decision, not an archaeology dig.

7. May's plea, answered — and what remains #

Let's revisit May's list, three months later:

1. "An official Mistral extension point for the permission layer" → delivered. Version 2.23's InteractionRequestBroker is exactly that: a public contract, typed events, idempotent resolution. The monkey-patch died a natural death, replaced by less code and more guarantees. Morality for everyone tinkering around an agent runtime: when upstream exposes the real API, rush to migrate — even if it starts with a crash.
2. "A public Vibe usage endpoint" → still nothing. The session context gauge works (context_tokens), the monthly credit gauge remains impossible without scraping. The public request stands.
3. "An official or community packaged kit" → still nothing on Mistral's side. The Claude ecosystem keeps growing (macropads, Stream Decks, official prototypes). The peripheral gap on the French stack remains wide open — which is precisely why the repo is Apache 2.0.

And on my own May roadmap: the voice button shipped (it's this article), the Fat Kitten took the "shake" easter egg's spot, and the rotary encoder is still waiting. That's the honest pace of an evenings-and-weekends project: one idea in four per release.

8. What's next #

What 2.23+ opens, already on the drawing board:

1. Answering the agent's questions with the buttons. Vibe sometimes asks genuine multiple-choice questions; the WaitingForInputEvent now carries predefined answers. The screen shows up to three options mapped to A/B/C — the device moves from "approve/reject" to "answering the agent." By far the biggest step toward the full remote control.
2. m5stack-gate: deterministic physical confirmation. Vibe's TOML hook system (pre_tool/post_tool) allows an official gate: until the button is pressed, the tool is denied by the runtime — no more "ABSOLUTE RULE" in natural language inside instructions, which depended on the model's obedience. Security policy becomes code, not rhetoric.
3. Real-time STT. Vibe embeds a streaming Voxtral client (websocket): live transcription while you speak, text live on the device's screen. No more dictating blind.
4. The ACP/app_server client, the royal road. Vibe is converging toward a client/server model (the 2.24 TUI itself became an app_server client, vibe-acp binary available). Rebuilding the bridge as a first-class ACP client would mean zero residual monkey-patching, resilience across version bumps — and compatibility with any ACP-speaking agent. That's the year-end foundational project.

Until then, the object lives its life on the kitchen table: I dictate, it executes, it answers — and the Fat Kitten watches over it all, impassive, on its rainbow background.

Repo #

Everything is there: github.com/rdelfosse/vibe-m5stack — Apache 2.0. The PlatformIO firmware, the Python plugin, the fifteen briefs (worth a workshop on agent-assisted development), the web flasher, and the full CHANGELOG from 0.4.0 to 0.6.0. Web flasher for the firmware, install.ps1 / install.sh for the plugin, vibe-m5stack doctor to diagnose. Fork, adapt, break, tell.

All cited sources verified as of 23 August 2026. Versions, dates and technical details come from the vibe-m5stack repo's CHANGELOG and code. Spot a factual error? Write to me at bonjour@romaindelfosse.fr — fixed within 72h, traced in a Git commit.