In this paper we establish a new connection between Tensor Network (tn)-constrained kernel machines and Gaussian Processes (gps). We prove that the outputs of Canonical Polyadic Decomposition (cpd) and Tensor Train (tt)-constrained kernel machines converge in the limit of large ranks to the same product kernel gp which we fully characterize, when specifying appropriate i.i.d. priors across their components. We show that tt-constrained models convergence faster to the gp compared to their cpd counterparts for the same number of model parameters. The convergence to the gp occurs as the ranks tend to infinity, as opposed to the standard approach which introduces tns as an additional constraint on the posterior. This implies that the newly established priors allow the models to learn features more freely as they necessitate infinitely more parameters to converge to a gp, which is characterized by a fixed learning representation and thus no feature learning. As a consequence, the newly derived priors yield more flexible models which can better fit the data, albeit at increased risk of overfitting. We demonstrate these considerations by means of two numerical experiments.

In today's data-driven landscape, the capacity to efficiently process and analyze vast datasets is crucial across various domains, including healthcare, climate modeling, and finance. Despite the growing need for scalable and interpretable machine learning models, traditional approaches, particularly kernel machines, face significant challenges due to the curse of dimensionality. As data complexity increases, the computational and memory demands of kernel machines often become prohibitive, limiting their applicability in high-dimensional applications. This thesis addresses these challenges by investigating the integration of tensor networks (TNs) with kernel machines, aiming to enhance scalability, efficiency without sacrificing predictive power and interpretability.

We propose that TNs, with their ability to represent high-dimensional data through low-rank structures, can effectively alleviate the limitations of kernel machines. Our research is structured around three central inquiries: first, we examine how TNs can accelerate kernel machine scalability while accurately approximating kernel functions; second, we elucidate the theoretical links between TN-constrained kernel machines and Gaussian processes, providing insights into convergence and generalization; finally, we introduce a novel optimization framework characterizing a specific TN, the multi-linear singular value decomposition (MLSVD), in terms of primal and dual problems.

Recent developments in wearable devices have made accurate and efficient seizure detection more important than ever. A challenge in seizure detection is that patient-specific models typically outperform patient-independent models. However, in a wearable device one typically starts with a patient-independent model, until such patient-specific data is available. To avoid having to construct a new classifier with this data, as required in conventional kernel machines, we propose a transfer learning approach with a tensor kernel machine. This method learns the primal weights in a compressed form using the canonical polyadic decomposition, making it possible to efficiently update the weights of the patient-independent model with patient-specific data. The results show that this patient fine-tuned model reaches as high a performance as a patient-specific SVM model with a model size that is twice as small as the patient-specific model and ten times as small as the patient-independent model.

In the context of kernel machines, polynomial and Fourier features are commonly used to provide a nonlinear extension to linear models by mapping the data to a higher-dimensional space. Unless one considers the dual formulation of the learning problem, which renders exact large-scale learning unfeasible, the exponential increase of model parameters in the dimensionality of the data caused by their tensor-product structure prohibits to tackle high-dimensional problems. One of the possible approaches to circumvent this exponential scaling is to exploit the tensor structure present in the features by constraining the model weights to be an underparametrized tensor network. In this paper we quantize, i.e. further tensorize, polynomial and Fourier features. Based on this feature quantization we propose to quantize the associated model weights, yielding quantized models. We show that, for the same number of model parameters, the resulting quantized models have a higher bound on the VC-dimension as opposed to their non-quantized counterparts, at no additional computational cost while learning from identical features. We verify experimentally how this additional tensorization regularizes the learning problem by prioritizing the most salient features in the data and how it provides models with increased generalization capabilities. We finally benchmark our approach on large regression task, achieving state-of-the-art results on a laptop computer.

For the first time, this position paper introduces a fundamental link between tensor networks (TNs) and Green AI, highlighting their synergistic potential to enhance both the inclusivity and sustainability of AI research. We argue that TNs are valuable for Green AI due to their strong mathematical backbone and inherent logarithmic compression potential. We undertake a comprehensive review of the ongoing discussions on Green AI, emphasizing the importance of sustainability and inclusivity in AI research to demonstrate the significance of establishing the link between Green AI and TNs. To support our position, we first provide a comprehensive overview of efficiency metrics proposed in Green AI literature and then evaluate examples of TNs in the fields of kernel machines and deep learning using the proposed efficiency metrics. This position paper aims to incentivize meaningful, constructive discussions by bridging fundamental principles of Green AI and TNs. We advocate for researchers to seriously evaluate the integration of TNs into their research projects, and in alignment with the link established in this paper, we support prior calls encouraging researchers to treat Green AI principles as a research priority.

The Hilbert–space Gaussian Process (hgp) approach offers a hyperparameter-independent basis function approximation for speeding up Gaussian Process (gp) inference by projecting the gp onto M basis functions. These properties result in a favorable data-independent O(M³) computational complexity during hyperparameter optimization but require a dominating one-time precomputation of the precision matrix costing O(NM²) operations. In this paper, we lower this dominating computational complexity to O(N M) with no additional approximations. We can do this because we realize that the precision matrix can be split into a sum of Hankel–Toeplitz matrices, each having O(M) unique entries. Based on this realization we propose computing only these unique entries at O(NM) costs. Further, we develop two theorems that prescribe sufficient conditions for the complexity reduction to hold generally for a wide range of other approximate gp models, such as the Variational Fourier Feature (vff) approach. The two theorems do this with no assumptions on the data and no additional approximations of the gp models themselves. Thus, our contribution provides a pure speed-up of several existing, widely used, gp approximations, without further approximations.

Kernel machines are one of the most studied family of methods in machine learning. In the exact setting, training requires to instantiate the kernel matrix, thereby prohibiting their application to large-sampled data. One popular kernel approximation strategy which allows to tackle large-sampled data consists in interpolating product kernels on a set of grid-structured inducing points. However, since the number of model parameters increases exponentially with the dimensionality of the data, these methods are limited to small-dimensional datasets. In this work we lift this limitation entirely by placing inducing points on a grid and constraining the primal weights to be a low-rank Canonical Polyadic Decomposition. We derive a block coordinate descent algorithm that efficiently exploits grid-structured inducing points. The computational complexity of the algorithm scales linearly both in the number of samples and in the dimensionality of the data for any product kernel. We demonstrate the performance of our algorithm on large-scale and high-dimensional data, achieving state-of-the art results on a laptop computer. Our results show that grid-structured approaches can work in higher-dimensional problems.