Sequential Monte Carlo method for training Neural Networks on non-stationary time series

Abstract

In this thesis, we study the sequential Monte Carlo method for training neural networks in the context of time series forecasting. Sequential Monte Carlo can be particularly useful in problems in which the data is sequential, noisy and non-stationary. We compare this algorithm against a gradient-based method known as stochastic gradient descent (SGD), a commonly used method for training neural networks. The performance of SGD on forecasting non-stationary, noisy time series can be poor due to the possibility of overfitting on the data. The sequential Monte Carlo method may offer a solution for the problems that arise in forecasting non-stationary time series with SGD neural networks. At the same time, neural networks trained with SGD give deterministic predictions, and there is a need for quantification of the uncertainty in the prediction. Sequential Monte Carlo sequentially samples the weights of the neural network, providing a posterior distribution on the weights and thus the outcome. In this work, the sequential Monte Carlo algorithm is tested and analyzed, with different parameter settings, on four time series to give an overview of the behavior. Furthermore, we apply the SMC algorithm on a convolutional neural network known as WaveNet. We show that the SMC algorithm is very well-suited for forecasting non-stationary time series, and can significantly outperform the gradient-based SGD method. Additionally, we show that for specific time series the SMC algorithm on a convolutional neural network outperforms the SMC algorithm on a fully-connected neural network.