Dynamics-based control offers a promising approach to exploring the motion potential of soft robots. However, inherently infinite degrees of freedom of these systems pose significant challenges for dynamics modeling, closely followed by the pressing robustness concerns arising from finite-dimensional approximations. This paper addresses these issues by proposing a physics-informed dynamics learning neural network and an adaptive fractional-order control for continuum soft robots. Specifically, a deep Lagrangian neural network is first developed with an embedded self-attention mechanism to enhance learning efficiency, accuracy, and data sensitivity. Subsequently, an adaptive fractional-order sliding mode controller is designed, leveraging the inherent historical memory properties of fractional calculus. This controller not only ensures robust shape control but also improves response speed and tracking accuracy. To further handle model discrepancies in the learned dynamics and external disturbances, a nonlinear disturbance observer is introduced to effectively estimate and compensate for lumped uncertainties, thereby ensuring reliable performance. Theoretical analysis confirms the closed-loop stability, while both simulation and experiment results validate the high dynamics fitting accuracy of the proposed network, as well as the robust and precise tracking capability of the fractional-order controller.