SurvivalRank: Decentralized Evolution of Search Models

Abstract

Google has held roughly 90% of the global search market for over a decade, and the obstacles to challenging it have multiplied. In the framework of Aghion and Howitt, economic growth arises from creative destruction, where new innovations displace old ones, and the cycle repeats. But creative destruction requires open entry, and in search, entry is prohibitively costly. To compete, a new entrant must crawl billions of pages, accumulate years of interaction data, and build infrastructure capable of serving results in milliseconds. Large language models have not restored competitive entry, and with training costs growing at approximately 2.4× per year, they have consolidated it further. This paper proposes SurvivalRank, a decentralized architecture designed to restore the entry condition that Aghion and Howitt take as given but that current search markets lack. Rather than having a single entity design the ranking algorithm, the system allows anyone to contribute a ranking model, distributed via BitTorrent. Peer-to-peer networks have previously demonstrated the feasibility of sharing and recommending content among strangers at scale. Each peer runs a multi-armed bandit that allocates queries across available models, shifting traffic toward those that perform better. Peers share evaluation evidence through a lightweight gossip protocol: every few seconds, each peer sends its latest observation for one random arm to one random stranger, using Lamport timestamps for deduplication. Models that fall sufficiently below the best performer are retired locally. When a superior model arrives, it displaces the incumbent and the cycle begins again. We evaluate SurvivalRank on five learning-to-rank benchmarks. A single peer converges to the oracle-best arm on every dataset, gossip cooperation closes a substantial fraction of the gap to the centralized oracle, and when a superior model is injected mid-run by a single peer, the network displaces the incumbent within 16–70 rounds after injection on all configurations. A Price's Equation decomposition confirms that a discrete innovation effect, a sustained positive selection effect, and a destruction effect from arm retirement are all active simultaneously, the empirical signature of Schumpeterian creative destruction operating without central coordination.