I Gave an LLM Control of a Real Drone

What I built

A quadcopter you talk to. “Hawk, take off.” “Go forward three metres.” “What do you see?” “Land.” Speech goes into a local Whisper model, a language model turns the sentence into a plan, and a validator written in plain Python decides whether each step is allowed before anything reaches the flight controller. The aircraft is a Holybro X500 V2 on a Pixhawk 6C, flying PX4, with a DJI Osmo Action 4 as its eyes.

I built it to find out what it actually takes to let a model act on the physical world without trusting it. The answer so far: the model is the easy part. Everything that matters is in the layer between the model and the motors.

How it works

1. Speech in. The microphone feeds Whisper running on the Mac’s GPU, about a tenth of a second per phrase. A small grammar catches the common orders (take off, move, land, stop, status) and flies them at once. Anything the grammar cannot parse goes to Claude with a strict set of tools: status, takeoff, move, yaw, creep, look, hold, land, return home. The model never produces MAVLink. It can only ask for one of those.
2. The validator. Every tool call passes through an executor that enforces hard limits in code: 5 m per move, 10 m ceiling, 20 m radius from home, a floor for horizontal moves, a minimum interval between motion commands. Motion is refused unless the aircraft is airborne with a valid position and fresh telemetry. Takeoff runs only if the pilot’s own words contained “take off”, “launch” or “lift off”. None of this lives in the prompt.
3. Flight. Approved steps become MAVSDK calls over the telemetry radio: a takeoff command, a reposition in Hold mode, a Land. Nothing streams. If the laptop process dies the aircraft simply holds, and every PX4 failsafe stays in force.
4. The pilot always wins. The executor watches the flight mode continuously. Any mode it did not command, Land, Return, Position, means a human has taken the sticks: every routine stops, nothing further is sent, and even a spoken “stop” is refused until the mode switch gives control back.
5. Eyes. A separate process owns the camera, runs a small YOLO detector on every frame, and serves frames over localhost. “What do you see” sends one frame with the question. With a flag, two consecutive sightings of a person put an airborne aircraft into Hold with no model involved. Vision never starts a manoeuvre.
6. Voice out. Hawk, a dry British flight engineer, reads back every manoeuvre from code and adds one line of its own. The read-back comes from the tool arguments, so you always hear direction and distance exactly once.

Lessons

• The first explanation that fits is not the explanation. On the first real session the aircraft climbed hard a metre from the ground and I hit kill. I blamed the battery failsafe. The flight log showed my own altitude top-up loop re-sending a Hold and a reposition after I had already selected Land. The fix was the override latch above, verified in the simulator by selecting Land mid-takeoff and mid-move.
• GPS near buildings lies with a straight face. Thirty satellites, a claimed vertical accuracy of 0.3 m, and an altitude that slid four metres in ten seconds. The barometer was honest throughout. Height reference is now the barometer.
• The radio is part of the control loop. A 915 MHz telemetry link in a city whose cellular network sits at 900 MHz drops acknowledgements. Commands are now judged by what the aircraft does, not by whether the ACK came back, and the link runs with error correction on.
• Latency is a design problem, not a model problem. Grammar-matched orders fly immediately; the model is asked in parallel only for a remark that plays while the aircraft is already moving.

Next flight

Fly it again with the barometer height reference and the battery failsafe set to Land, on open ground, with the kill switch in hand. Then the camera goes on the airframe, and the next question is whether vision can be allowed to steer anything at all.

Hardware

Build it yourself

What I'm building next