Bayesian Tensor Train Kernel Machine

Abstract

Kernel machines are a class of machine learning models which are powerful in terms of predictive power, but are limited to low dimensional data, as their computational complexity scales according to O(N 3). Kernel methods are able to reduce this computational scaling to be linear in D by assuming a low rank structure on the weights vector. Existing tensor network kernel machines operate mostly in a deterministic setting and lack uncertainty quantification.
On top of this, hyperparameter tuning for these types of models can be costly and complex. A fully Bayesian kernel machine, employing a CP decomposition has been shown to perform well on regression and classification tasks without additional computational cost. hyperparameter tuning was aided for this model through the introduction of ARD priors. This model is still limited to a relatively small data dimension D, due to numerical instabilities caused by repeated Hadamard products in the CPD formulation. Alternative tensor decompositions such as the tensor train decomposition exist that do not rely on Hadamard products in their formulation and may therefore be more numerically stable.
This thesis proposes a Bayesian Tensor Train Kernel Machine, a fully Bayesian tensor train decomposed kernel machine. The proposed model will include ARD hyperpriors to aid in model selection, these priors allow for automatic inference of model complexity, as well as increasing model interpretability. Mean-field variational inference is used to approximate posterior distributions of all model parameters and hyperparameters. Experiments on synthetic datasets show the functioning of the ARD hyperpriors, and the increased numerical stability compared to the CP based model. Experiments on real world datasets showcases the performance of the proposed model in terms of prediction accuracy and uncertainty quantification compared to currently available models.