← all posts site ↗

saying flashlight turns on the flashlight

the phone assistant finally does real device actions, and the architecture is embarrassingly simple. speech converts locally, the text goes into the session, the agent replies with a function keyword instead of a paragraph, and the app maps that keyword to an action running on the phone itself. no cloud in the middle of the action, no waiting for a sentence to come back, the model just names the thing and the phone does the thing. i was sure android must have something like this and it turns out the trick is that the pattern was always the hard part, not the plumbing.

said flashlight, the flashlight turned on. asked for an app, it opened with a once or always prompt so it stops asking. the gaps are the honest ones: asked to play a song and it opened the music app without playing anything, and a video request opened youtube without the search, so the next layer is deep linking, actually landing inside the song and running the search instead of just launching the app. the keyword mapping generalizes to everything once you treat the model as a namer of actions, not a doer of them.

shipped a couple of versions in the same night with the sessions widget and the wake model baked in. one gotcha took a while: the phone's session profile only allowed a minimal toolset, so the new toolset couldn't reach the agent sessions at all until it was allowed through, and an update also went gray screen on first launch which ate a chunk of the evening. but the core loop works now, say it, get a keyword, phone executes locally, and that loop turns any sentence into a button.