This article presents an adaptive control strategy to stabilize a micro quadrotor in all three axes using only an Inertial Measurement Unit (IMU) for the attitude control and a monocular camera for canceling position drift. The proposed control scheme automatically determines the appropriate optical flow control gains. This is achieved by extending the stability-based approach to distance estimation developed in [1] to allow for the control of all three axes of a quadrotor. An analysis is done in simulation to present a proof of concept of the stabilization method and to determine the effects of scaling. Furthermore we verify the effects of varying effective camera frame rates and investigate how this control approach generalizes to smaller drone sizes. Actual flight tests are then performed on a Parrot ARDrone 2.0 and on a Parrot Bebop to show that both quadrotors achieve stable hover without position drift using only their IMU and bottom camera.