← all posts site ↗

the search stack that failed silently

the web stuff had been quietly dying and nothing was telling me. page extraction pointed at a provider this build doesn't even ship, so every single page read failed. search went through a local aggregator whose only engine was duckduckgo, and duckduckgo's scraper returned zero results, and the app reported that as ok. so everything looked healthy while everything was empty. silent failure is the worst kind, there is nothing to debug until you actually try to rely on it.

fixed extraction by pointing it at a provider that works. for search, installed a batch of engines from the official repo. bing works, brave works and returns real results, google and ecosia report blocked from this ip, startpage serves a captcha, and those three need a proxy transport before they're useful. the duckduckgo zero was the mystery though. it turned out the ip gets served an anti-bot anomaly page: plain requests come back with zero results and a status the code reads as success. the workaround is to grab the vqd token from the search page first and then post the query form, which is what the official client does.

patched that flow in with one retry and a fallback to the old get request. fresh queries came back with ten results, four for four. also learned the store overwrites engine files on update, so the patch lives in a pending patches folder and gets re-applied after updates. and i hammered brave with test queries hard enough to trip its rate limit, which at least proves the rate limit does something. the lesson that stays: an empty result with a 200 feels identical to a healthy one unless you actually look.