Continuous state and action Q-learning framework applied to quadrotor UAV control

Abstract

This paper describes an implementation of a reinforcement learning-based framework applied to the control of a multi-copter rotorcraft. The controller is based on continuous state and action Q-learning. The policy is stored using a radial basis function neural network. Distance-based neuron activation is used to optimize the generalization algorithm for computational performance. The training proceeds off-line, using a reduced-order model of the controlled system. The model is identified and stored in the form of a neural network. The framework incorporates a dynamics inversion controller, based on the identified model. Simulated flight tests confirm the controller's ability to track the reference state signal and outperform a conventional proportional-derivative(PD) controller. The contributions of the developed framework are a computationally-efficient method to store a $\mathcal{Q}$-function generalization, continuous action selection based on local $\mathcal{Q}$-function approximation and a combination of model identification and offline learning for inner-loop control of a UAV system.