P. Johari
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
2 records found
1
Scaling Laws for Compound AI Systems
More Compute, More Problems? Rethinking Scaling Laws for Retrieval-Augmented Generation
Retrieval-Augmented Generation (RAG) extends Large Language Models with external knowledge retrieval, yet the scaling behavior of inference-time compute in multi-stage RAG pipelines remains poorly characterized. This thesis provides a systematic empirical investigation of test-time compute scaling along three axes: retrieval depth (expanding the candidate pool size c presented to a neural reranker), query breadth (generating n LLM-produced query reformulations and fusing their results), and adaptive budget allocation (dynamically distributing cross-encoder compute via ORE and ReformIR). Experiments span three benchmark datasets (MS-MARCO, TriviaQA/DPR-w100, 2WikiMultiHopQA), two first-stage retrievers (BM25, TCT-ColBERT), six fusion algorithms, and two pipeline orderings (merge-then-rerank vs. rerank-then-merge).
Our findings establish six empirical principles for test-time compute allocation in compound retrieval systems. First, depth scaling yields monotonic gains only when the reranker is architecturally robust to noise; the generative reranker MonoT5 degrades at large c while the distilled cross-encoder MiniLM-L6 scales reliably. Second, query breadth scaling requires a continuous embedding space, dense retrievers enable substantially larger fusion gains than sparse retrievers because semantically distinct rewrites activate genuinely different document regions. Third, the optimal pipeline ordering depends on the reranker's domain calibration: rerank-then-merge excels in-domain, while voting-based fusion dominates out-of-domain. Fourth, adaptive and fusion-based scaling are complementary rather than competing. Fifth, a persistent 15-17% relative Recall@50 Oracle gap reveals that practical fusion algorithms, not candidate generation, constitute the performance bottleneck. Sixth, retrieval gains transfer only partially to downstream generation quality, with volatile LLM-as-Judge trajectories cautioning against assuming proportional end-to-end improvements.
These results demonstrate that, unlike the monotonic scaling laws governing language model pre-training, retrieval scaling is non-monotonic and conditional, effective test-time compute allocation in Compound AI Systems requires domain-aware orchestration across the full retriever-reranker-fusion-reader pipeline. ...
Our findings establish six empirical principles for test-time compute allocation in compound retrieval systems. First, depth scaling yields monotonic gains only when the reranker is architecturally robust to noise; the generative reranker MonoT5 degrades at large c while the distilled cross-encoder MiniLM-L6 scales reliably. Second, query breadth scaling requires a continuous embedding space, dense retrievers enable substantially larger fusion gains than sparse retrievers because semantically distinct rewrites activate genuinely different document regions. Third, the optimal pipeline ordering depends on the reranker's domain calibration: rerank-then-merge excels in-domain, while voting-based fusion dominates out-of-domain. Fourth, adaptive and fusion-based scaling are complementary rather than competing. Fifth, a persistent 15-17% relative Recall@50 Oracle gap reveals that practical fusion algorithms, not candidate generation, constitute the performance bottleneck. Sixth, retrieval gains transfer only partially to downstream generation quality, with volatile LLM-as-Judge trajectories cautioning against assuming proportional end-to-end improvements.
These results demonstrate that, unlike the monotonic scaling laws governing language model pre-training, retrieval scaling is non-monotonic and conditional, effective test-time compute allocation in Compound AI Systems requires domain-aware orchestration across the full retriever-reranker-fusion-reader pipeline. ...
Retrieval-Augmented Generation (RAG) extends Large Language Models with external knowledge retrieval, yet the scaling behavior of inference-time compute in multi-stage RAG pipelines remains poorly characterized. This thesis provides a systematic empirical investigation of test-time compute scaling along three axes: retrieval depth (expanding the candidate pool size c presented to a neural reranker), query breadth (generating n LLM-produced query reformulations and fusing their results), and adaptive budget allocation (dynamically distributing cross-encoder compute via ORE and ReformIR). Experiments span three benchmark datasets (MS-MARCO, TriviaQA/DPR-w100, 2WikiMultiHopQA), two first-stage retrievers (BM25, TCT-ColBERT), six fusion algorithms, and two pipeline orderings (merge-then-rerank vs. rerank-then-merge).
Our findings establish six empirical principles for test-time compute allocation in compound retrieval systems. First, depth scaling yields monotonic gains only when the reranker is architecturally robust to noise; the generative reranker MonoT5 degrades at large c while the distilled cross-encoder MiniLM-L6 scales reliably. Second, query breadth scaling requires a continuous embedding space, dense retrievers enable substantially larger fusion gains than sparse retrievers because semantically distinct rewrites activate genuinely different document regions. Third, the optimal pipeline ordering depends on the reranker's domain calibration: rerank-then-merge excels in-domain, while voting-based fusion dominates out-of-domain. Fourth, adaptive and fusion-based scaling are complementary rather than competing. Fifth, a persistent 15-17% relative Recall@50 Oracle gap reveals that practical fusion algorithms, not candidate generation, constitute the performance bottleneck. Sixth, retrieval gains transfer only partially to downstream generation quality, with volatile LLM-as-Judge trajectories cautioning against assuming proportional end-to-end improvements.
These results demonstrate that, unlike the monotonic scaling laws governing language model pre-training, retrieval scaling is non-monotonic and conditional, effective test-time compute allocation in Compound AI Systems requires domain-aware orchestration across the full retriever-reranker-fusion-reader pipeline.
Our findings establish six empirical principles for test-time compute allocation in compound retrieval systems. First, depth scaling yields monotonic gains only when the reranker is architecturally robust to noise; the generative reranker MonoT5 degrades at large c while the distilled cross-encoder MiniLM-L6 scales reliably. Second, query breadth scaling requires a continuous embedding space, dense retrievers enable substantially larger fusion gains than sparse retrievers because semantically distinct rewrites activate genuinely different document regions. Third, the optimal pipeline ordering depends on the reranker's domain calibration: rerank-then-merge excels in-domain, while voting-based fusion dominates out-of-domain. Fourth, adaptive and fusion-based scaling are complementary rather than competing. Fifth, a persistent 15-17% relative Recall@50 Oracle gap reveals that practical fusion algorithms, not candidate generation, constitute the performance bottleneck. Sixth, retrieval gains transfer only partially to downstream generation quality, with volatile LLM-as-Judge trajectories cautioning against assuming proportional end-to-end improvements.
These results demonstrate that, unlike the monotonic scaling laws governing language model pre-training, retrieval scaling is non-monotonic and conditional, effective test-time compute allocation in Compound AI Systems requires domain-aware orchestration across the full retriever-reranker-fusion-reader pipeline.
Learning Curve Extrapolation using Machine Learning
Benefits and Limitations of using LCPFN for Learning Curve Extrapolation
This study explores the extrapolation of learning curves, a crucial aspect in evaluating learner performance with varying dataset sample sizes. We use the Learning Curve Prior Fitted Network (LC-PFN), a transformer pre-trained on synthetic data with proficiency in approximate Bayesian inference, to investigate its predictive accuracy using the Learning Curve Database (LCDB). The assessment involves MSE as an error metric, with 2 baselines from previous studies where we see it outperform the baseline in some cases and keep on par in others. Additionally, we scrutinize instances where the LC-PFN model may exhibit shortcomings to identify trends in curve extrapolation failures, offering insights for potential modifications to the training dataset. We see a pattern in learners where LC-PFN performs consistently poorly on, whereas no significant pattern can be seen for datasets.
...
This study explores the extrapolation of learning curves, a crucial aspect in evaluating learner performance with varying dataset sample sizes. We use the Learning Curve Prior Fitted Network (LC-PFN), a transformer pre-trained on synthetic data with proficiency in approximate Bayesian inference, to investigate its predictive accuracy using the Learning Curve Database (LCDB). The assessment involves MSE as an error metric, with 2 baselines from previous studies where we see it outperform the baseline in some cases and keep on par in others. Additionally, we scrutinize instances where the LC-PFN model may exhibit shortcomings to identify trends in curve extrapolation failures, offering insights for potential modifications to the training dataset. We see a pattern in learners where LC-PFN performs consistently poorly on, whereas no significant pattern can be seen for datasets.