i wanted a pair of smart glasses that work like an open platform. put them on, press a button in an app to start listening, and then talk naturally. the glasses capture what i say, optionally snap a photo of what i am looking at, send both to my ai agent, and speak the response back through the phone. no wake word, no cloud dependency for the speech recognition, no button pressing between questions. just a continuous back and forth.
the meta ray-ban gen 1 glasses cannot do any of this natively. the wake word is firmware-locked, the capture button is hardwired to meta ai, and there is no public api for intercepting either. but the dat sdk does expose the camera stream and mic audio while a session is active, and that is enough to build around. the rest of the pipeline runs on the phone and on a home server.
the voice activity detection runs entirely on-device using android's built-in speech recognizer. no network calls, no model downloads. when the glasses are paired as a bluetooth headset, android automatically routes the bt mic to the speech recognizer through standard hands-free profile. the recognizer streams partial results in real time. when it detects about one point two seconds of silence after speech, it finalizes the transcript and fires the query. this avoids the problem of having to press a button for every question. you talk, pause, and the system responds.
the camera is not always on. the speech recognizer scans partial transcripts for visual trigger words. look, see, what is this, describe, read this, picture, snap. if any of them appear, the app spins up a dat camera session in the background while you are still finishing your sentence. the two-to-three-second session negotiation happens during the tail of your speech, so by the time the recognizer returns the final transcript, the camera is ready to capture. if no visual trigger word is detected, the photo is skipped entirely and the query is text-only. this saves battery compared to keeping the camera streaming continuously.
the text and optional photo hit a bridge running on my home server as a systemd service. the bridge routes simple conversational queries directly to a fast multimodal model over openrouter, bypassing the full agent stack for sub-second latency. queries that need tools, web search, home assistant integration, or image analysis go through the full ai agent with a persistent session. the response comes back and gets piped to a local kokoro tts server running on a windows machine, which synthesizes the audio and streams it back to the phone. the phone plays it through mediaplayer on the media audio stream so it comes through whatever the active audio output is, which when the glasses are paired as a headset means the glasses speakers.
the loop restarts automatically after every response. the speech recognizer picks back up, ready for the next question. it is not the seamless magic of a built-in assistant, but it is an open pipeline where every component is replaceable. the model, the tts engine, the trigger words, the routing logic, all of it. that is the thing meta will never give you, and it is worth building.