the bigger version of the idea: scrape every game worth knowing from three storefronts into one searchable corpus. steam through a batched store endpoint, epic through its graphql edge, itch through its xml feed. every game needs description, tags, rating, genre, release date, budget tier, review count, price. the crawlers had to be resumable, rate limited, and patient, they run for days and shouldn't die if one page breaks.
the numbers so far: over eighty thousand apps in the steam catalog, tens of thousands of epic offers, thousands of itch games, and a steam sweep still walking the store at a few requests a second. a local web app already serves the corpus for scanning, searching, filtering, ranking, and prioritizing, reachable from the phone. the first real analysis fires automatically once the corpus crosses twelve thousand games, so i can wake up to a site instead of a status message.
everything runs incremental by design. a crawl that gets killed midway restarts where it left off, no state lost. the unifying move is the same as the smaller prototype: embed everything, build the space, then find the distant points and ask a model to translate them back into concepts. the difference is scale and the guardrails that make scale possible.