In this paper, we recast the variational formulation corresponding to the single layer boundary integral operator V for the wave equation as a minimization problem in L2(Σ), where Σ:=∂Ω×(0,T) is the lateral boundary of the space-time domain Q:=Ω×(0,T). For discretization, the minimization problem is restated as a mixed saddle point formulation. Unique solvability is established by combining conforming nested boundary element spaces for the mixed formulation such that the related bilinear form is discrete inf-sup stable. We analyze under which conditions the discrete inf-sup stability is satisfied, and moreover, we show that the mixed formulation provides a simple error indicator, which can be used for adaptivity. We present several numerical experiments showing the applicability of the method to different time-domain boundary integral formulations used in the literature.