Aerial physical interaction opens the door for many operations at height to be automatised using aerial robots. This research presents a novel manipulator design mounted on a traditional quadrotor, which utilises both mechanical and software compliance to perform physical interaction on vertical walls and overhanging surfaces, such as those found under bridges. A centralised impedance control scheme allows direct control of the end-effector pose without needing separate modes for free-flight and contact. A spring-loaded prismatic joint provides passive compliance while doubling as a force-feedback for the impedance controller through measuring the spring displacement. Simulation and flight experiments prove the feasibility and robustness of this approach for exchanging high forces at height, with a total of 44 successful experiments carried out in four sets. An average maximum force of 5.66 N or 19.3\% of the system's weight was achieved over one set of 11 experiments.