Scaling Laws for Compound AI Systems

Abstract

Retrieval-Augmented Generation (RAG) extends Large Language Models with external knowledge retrieval, yet the scaling behavior of inference-time compute in multi-stage RAG pipelines remains poorly characterized. This thesis provides a systematic empirical investigation of test-time compute scaling along three axes: retrieval depth (expanding the candidate pool size c presented to a neural reranker), query breadth (generating n LLM-produced query reformulations and fusing their results), and adaptive budget allocation (dynamically distributing cross-encoder compute via ORE and ReformIR). Experiments span three benchmark datasets (MS-MARCO, TriviaQA/DPR-w100, 2WikiMultiHopQA), two first-stage retrievers (BM25, TCT-ColBERT), six fusion algorithms, and two pipeline orderings (merge-then-rerank vs. rerank-then-merge).

Our findings establish six empirical principles for test-time compute allocation in compound retrieval systems. First, depth scaling yields monotonic gains only when the reranker is architecturally robust to noise; the generative reranker MonoT5 degrades at large c while the distilled cross-encoder MiniLM-L6 scales reliably. Second, query breadth scaling requires a continuous embedding space, dense retrievers enable substantially larger fusion gains than sparse retrievers because semantically distinct rewrites activate genuinely different document regions. Third, the optimal pipeline ordering depends on the reranker's domain calibration: rerank-then-merge excels in-domain, while voting-based fusion dominates out-of-domain. Fourth, adaptive and fusion-based scaling are complementary rather than competing. Fifth, a persistent 15-17% relative Recall@50 Oracle gap reveals that practical fusion algorithms, not candidate generation, constitute the performance bottleneck. Sixth, retrieval gains transfer only partially to downstream generation quality, with volatile LLM-as-Judge trajectories cautioning against assuming proportional end-to-end improvements.

These results demonstrate that, unlike the monotonic scaling laws governing language model pre-training, retrieval scaling is non-monotonic and conditional, effective test-time compute allocation in Compound AI Systems requires domain-aware orchestration across the full retriever-reranker-fusion-reader pipeline.