Dynamic Airspace Reconfiguration with Deep Reinforcement Learning

Abstract

For future operations of unmanned aviation, even higher traffic densities than previously seen in manned aviation are expected. Previous work has shown that a vertically layered airspace design performs best at improving safety metrics such as the total number of conflicts and Losses of Separation (LoSs). Furthermore, it has been shown that machine learning techniques are capable of selecting heading ranges for the vertically stacked layers in non-uniform traffic scenarios, in order to reduce the number of conflicts and LoSs compared to uniform structures. These works, however, set structures in an ‘empty’ airspace and do not take into account the necessary vertical deviations to get from one structure to the next. In this work reinforcement learning (RL) agents are used to select layer heading ranges, while taking into account the previous airspace structure. During this dynamic structuring, several challenges arise. First, it is not clear how to reduce the number of vertical conflicts when aircraft move into a new airspace structure. Second, specific structures should be selected that reduce the necessary vertical deviations from the old structure, while still minimising the cruising conflicts for the new traffic distribution. The present work is divided into three experiments. Experiment I focused on analysing the number of conflicts and LoSs that aircraft suffer during vertically moving towards their layer in the new structure. Experiment II tested whether a RL agent is capable of setting an aircraft structure in function of the expected future traffic scenario. Experiment III aimed to show the capability of a RL agent to select airspace structures, while taking into account the previous airspace structures, in order to decrease the number of vertical conflicts. The results of the research show that RL methods are capable of defining airspace structures appropriate for a given traffic scenario. For dynamic reconfiguration, it proved challenging to simulate traffic scenarios that cause an agent to select different structures to prevent the occurrence of vertical conflicts. Under the experimental conditions employed, analytical methods of structure selection performed better in terms of safety.