← all posts site ↗

say play and the app just opens

the basic phone commands worked, but ask for actual depth and the limits showed fast. say play a song and the music app opens and nothing plays. say play a video and it searches the browser instead of using the youtube app. the ask for day two was a hard cap on real capabilities: wifi, bluetooth, airplane mode, auto rotate, dark mode, tailscale, now playing to identify a song, and the big ones, play a specific song, play the liked playlist, search a video, all inside the real apps, not the browser.

the first round failed honestly. whatsapp did not open, a nest group did not open, auto rotate went to the modify settings grant screen and the app was not even listed there, and the music app still refused to play anything. saying the youtube fork works when it demonstrably does not was exactly the kind of claiming that had to stop, so the rule became: only claim what a real check can prove.

the response was to stop pretending and take the wheel. phone unlocked, wireless adb still on, so the agent drove the phone directly, screenshots, taps, natural language commands turned into actions. the first real win was a video actually playing with the queue loaded, which was the top ask working end to end. then two bugs got caught in the mess: a refresh call that returns a boolean instead of the node it looked like it should, and an accessibility tree that lives in a different window during transitions, which is why the service could not see what the ui dumper could. scanning all windows fixed the second.

the last honest touch: a play verification check that reads whether audio is actually active, so the speech only says playing when it really is. the proven flow got restored with the fixes kept, and the failures list became the roadmap instead of the shame pile.