Tensor Network-Constrained Kernel Machines as Gaussian Processes

Abstract

In this paper we establish a new connection between Tensor Network (tn)-constrained kernel machines and Gaussian Processes (gps). We prove that the outputs of Canonical Polyadic Decomposition (cpd) and Tensor Train (tt)-constrained kernel machines converge in the limit of large ranks to the same product kernel gp which we fully characterize, when specifying appropriate i.i.d. priors across their components. We show that tt-constrained models convergence faster to the gp compared to their cpd counterparts for the same number of model parameters. The convergence to the gp occurs as the ranks tend to infinity, as opposed to the standard approach which introduces tns as an additional constraint on the posterior. This implies that the newly established priors allow the models to learn features more freely as they necessitate infinitely more parameters to converge to a gp, which is characterized by a fixed learning representation and thus no feature learning. As a consequence, the newly derived priors yield more flexible models which can better fit the data, albeit at increased risk of overfitting. We demonstrate these considerations by means of two numerical experiments.