Sparse-Exploration Reinforcement Learning for Control of Quantum Error Correction

Abstract

Quantum error correction (QEC) needs a quantum computer's control parameters to stay calibrated as a program runs, but these parameters drift on minute-to-hour timescales, requiring periodic recalibration. Sivak et al. propose a reinforcement-learning agent that re-tunes them online, using QEC detection events as its reward and avoiding pauses for recalibration. We independently replicate their simulation and policy-gradient (PGPE) agent, and propose sparse exploring: rather than perturbing every control parameter in each sample, the agent perturbs only a random subset, lowering the mean error rate while it learns. The price of learning, referred to as the exploration gap, grows with the code distance (the size of the error-correcting code) and as the irreducible error rate (the error floor of perfectly tuned hardware) falls, and it compounds over program length. It is therefore expected to matter more on future hardware, and sparse exploring shrinks it. In simulation, under both idealized sinusoidal and hardware-inspired band-limited 1/f drift, sparse exploring closes roughly 74-85% of the exploration gap, reaching up to 90% for slow drift, independent of code distance (tested up to d=9) and irreducible error rate. We give both a fixed-k variant and an adaptive estimator that picks the sparsity online; the adaptive variant tracks steady drift well but underperforms under sudden fast drift. The result is a cheap addition to online steering that lowers error rates over the multi-day programs useful algorithms require, pending confirmation on real hardware.