Improving the performance of Recurrent Neural Networks for time series prediction by combining Long Short-Term Memory and Attention Long Short-Term Memory

Abstract

Recurrent neural networks (RNNs) used in time series prediction are still not perfect in their predictions and improvements can still be made in the area. Most recently transformers have led to great improvements in the field of RNNs, however transformers can not be used on time series data, because the architecture of transformers does not account for the flow of time and would use future data to predict past events. This research aims to further improve the performance of machine learning models on time-series prediction. It attempts to do so by implementing a new neural network model based on the multi-head attention mechanism (used in transformers) and combining it with an already existing neural network model called long short term memory (LSTM). To test whether the newly implemented models have improved performance they are tested on a weather dataset and compared on their ability to correctly predict daily maximum temperatures. The final results however show that combining LSTM and ALSTM models does not results in an improved loss that is worth the extra instability that is added to the model and the extra computational cost that is needed to train the model.