wanted to replace meta ai on my ray-ban glasses with my own agent so when i tap or ask something it goes to me instead of meta's cloud. found a project called visionclaw that already solved the hard parts — they built an android app using meta's dat sdk that streams glasses camera and mic to gemini live api and then delegates tool calls to a gateway. i don't want the streaming or gemini though. just want tap to capture a photo, maybe say something, send it to my server, get an answer back as audio through the glasses speaker.
the simplest version is an mvp where the phone app sits in background, connected to the glasses via ble. when triggered — either by a phone widget or intercepting the glasses button — it captures the current frame and records a few seconds of audio. posts both as a single http request to my server. i do the vision analysis and speech-to-text on the server side, reason over it, generate a text response, convert that to speech, and send the audio file back. the phone plays it through the glasses. one round trip, no continuous streaming, no wake word listening in the cloud. latency is probably three to six seconds which is fine for "what am i looking at" questions but not for conversation.
the hard parts were already solved by visionclaw — they got the dat sdk working, figured out the pairing, frame capture, and button handling. the remaining challenges are battery (keeping a background process alive without draining the phone), audio routing (getting the response to play through the glasses speaker instead of the phone speaker), and wake word detection (the glasses button normally triggers meta ai, so i'd need to intercept it or use a phone widget instead). the architecture ends up being phone captures and sends, server processes everything, audio comes back. server side is basically just one http endpoint that receives multipart jpeg plus audio, runs whisper for transcription, feeds the image and text to the agent, tts the response, returns audio. android side needs the dat sdk connection, capture trigger, http client, and audio playback. it's doable but the android app is the piece that doesn't exist yet and would need android studio to build.