Haptic interaction is critical in physical Human-Robot Interaction (pHRI), given its wide applications in manufacturing, medical and healthcare, and various industry tasks. A stable haptic interface is always needed while the human operator interacts with the robot. Passivity-based approaches have been widely utilised in the control design as a sufficient condition for stability. However, it is a conservative approach which therefore sacrifices performance to maintain stability. This paper proposes a novel concept to characterise an ultimately passive system, which can achieve the boundedness of the energy in the steady-state. A so-called Ultimately Passive Controller (UPC) is then proposed. This algorithm switches the system between a nominal mode for keeping desired performance and a conservative mode when needed to remain stable. An experimental evaluation on two robotic systems, one admittance-based and one impedance-based, demonstrates the potential interest of the proposed framework compared to existing approaches. The results demonstrate the possibility of UPC in finding a more aggressive trade-off between haptic performance and system stability, while still providing a stability guarantee.