Structural health monitoring aims to ensure the reliable operation of structures through the use of sensors and data analysis. It is important to develop methods that not only detect damage, but also provide a measure of uncertainty, allowing engineers to make well-informed decisions about maintenance. Recently, deep learning-based amortized Bayesian inference methods have emerged as promising tools for probabilistic structural health monitoring. One such method is BayesFlow, which can theoretically enable efficient damage detection, including uncertainty quantification, directly from sensor data without requiring explicit likelihood functions. However, its practical application is hindered by the high computational cost of generating a large and diverse set of training simulations. This work proposes a method to address the problematic simulation data requirements, thereby increasing the feasibility of BayesFlow as a viable tool for structural health monitoring. A physics-informed approach is developed that enhances the summary network component of BayesFlow by integrating knowledge about the structure and sensor network. Specifically, a Graph Summary Neural Network is designed, leveraging the sequential nature of sensor measurements and the spatial topology of the sensor network. Through a progressive case study based on a model truss bridge, it is demonstrated that the GSNN surpasses baseline networks in terms of both efficiency and performance, thus bringing BayesFlow-based SHM closer to real-world deployment.