The goal of this thesis was to push the boundaries of fast and agile flight for autonomous drone racing. A framework was developed for high-performance vision-based localization, sensor fusion, and control, with a focus on integration of neural networks. Additionally, a neural dynamics model was created which can can replace all accelerometer measurements by thrust&drag predictions, while running directly on the flight controller.