Hawk learns to talk

Written from the day’s logs.

The pilot is called Hawk. A dry British flight engineer: calm, precise, amused, never a comedian. I wanted something I could show a friend and have them grin. A flat robotic acknowledgement does not do that.

The pipeline

Microphone to Whisper, running locally on the Mac’s GPU, about a tenth of a second per phrase. Then a small grammar: take off, go left five, land, return home, status. If the grammar understands the order, it is queued and flown immediately. If not, Claude gets it with a strict set of tools and the system prompt that makes it Hawk.

Then speech out. macOS say at first, then ElevenLabs once the keys were in. The voice I actually wanted needs a paid tier I do not have, so Hawk sounds like George for now.

Everything that was wrong by lunchtime

“I told it left and it moved right.” It had not. I had typed “right” in the test script. But the confusion underneath was real: left and right relative to the aircraft’s nose look backwards when you watch from behind it facing the other way. So there is now a flag that pins “forward” to a compass heading. For a demo, forward is whichever way the audience faces.

“It keeps saying I’m one metre short.” The executor was reporting the residual after every move. PX4 repositions settle within about a metre by design. Nobody needs to hear about it. Now it says “arrived” and logs the rest.

“There’s a lot of latency.” Every utterance was waiting for the model before anything moved. Now grammar-matched commands fly at once, the mic reopens immediately, and the model is asked in parallel for a one-line remark that plays while the aircraft is already moving. The read-back of direction and distance comes from code, not from the model, so you hear it once and it is always right.

“I get the announcement twice.” The model was repeating what the code had already said. One rule in the prompt: the manoeuvre has been read back, do not repeat it.

“It doesn’t understand ‘land’.” Single short words are Whisper’s weakest case. A bigger model, a vocabulary prompt and a fuzzy snap for one-word orders fixed most of it.

The honest moment

At one point Hawk told the model’s conversation history that it had never actually flown anything, because the fast path was bypassing the model and the history had no record of tools firing. The fix was a flight-log line written into the conversation after every executed command. The model should not have to guess what the aircraft did.

By the end of the day: takeoff, moves in both frames, yaw, a slow creep until I say stop, return, land, all in the simulator, all by voice, with Hawk saying things like “Mind the furniture, sir.” Good enough to show someone.