Most physical systems of interest are chaotic in nature. Quick and reasonably accurate solutions for these systems are essential to various fields such as the effective control mechanism construction and early-stage design. However, their chaotic nature also leads to them being computationally expensive to model using traditional numerical techniques. Reduced Order Modelling (ROM) is an umbrella term that describes any method wherein a system state is projected into a ‘simplified’ state space in order to extract meaningful information about its dominant components. Traditional ROMs whilst effective in capturing statistical information, tend to neglect smaller-energy components responsible for short-term dynamics. With the advances seen in data-driven approaches over the past decade, namely in machine learning (specifically deep learning), these methods seem to offer attractive alternatives to traditional ROM techniques and are the focus of this thesis.
It is found that in existing literature there is no consensus on which ML models to use and no real extensive comparative studies either. Further, there is no unified approach to training the different RNN models, and only one surveyed study utilises an auto-regressive optimization strategy. This is a major pitfall, since the RNNs are only trained for single-step prediction but tested on multi-step prediction - a clear contradiction in the training and testing objectives. Additionally, no studies have been performed on true Runge-Kutta methods-inspired layering architectures, which show initial promise in increasing prediction horizons at little additional compute cost. As such, an extensive comparative study is performed, pitting long-short term memory networks, gated recurrent unit networks and echo state networks against each other. Further, the effect of RK-inspired layering is tested against a standard single-layer network. The effect of incorporating contractive losses in the auto-encoders is also investigated, in addition to multi-regime autoencoders. Every model is tested on chaotic systems of increasing complexity, namely the Lorenz ’63 system, the Charney-DeVore system, the Kuramoto-Sivashinsky system and the Kolmogorov flow system.
It is observed that the contractive loss does not help in modelling a smoother latent space, and instead leads to the unintended minimization of the norm of the latent space variables. Multi-regime auto-encoders are found to work quite well, with well separated latent spaces reflecting the nature of the dynamics (chaotic/periodic). Auto-regressive training is found to be crucial for increasing the prediction performance of back-prop trained RNNs (provided additional care is taken to limit their gradients during training) while it has little to no effect on the ESNs. Further, an initial skip-layer is found to be beneficial whereas, higher order RK layering architectures provide diminishing to no returns.