Incremental urban and community expansion in rural heritage landscapes often produces cumulative visual impacts, yet planning rarely specifies a clear endpoint for acceptable change. This paper proposes an integrated Visual Impact Assessment (VIA) framework, aligned with SDG 11, to determine “when to stop” using stage-comparable evidence across past, present, and future conditions. The framework is organized in three modules. First, a point cloud-enhanced GIS module quantifies visibility and spatial change across development stages. Second, an enhanced Key Observation Point (KOP) module derives matched eye-level evidence from multi-temporal street-level panoramas and scenario visualizations, for example using Street View Imagery (SVI) time series and 3D Gaussian Splatting (3DGS) rendering. Third, a decision layer integrates structured public acceptability from a questionnaire covering different respondent groups with in-depth expert interviews and synthesis, with virtual reality (VR) eye- and head-tracking used as supportive behavioral evidence. Applied to the Middenbeemster expansion in the Beemster Polder, the Netherlands, the framework yields a case-calibrated reference package for decision support: KOP-based construction intensity serves as the primary reference line for review, perception indicators serve as supporting guardrails, spatial character metrics act as case-specific reference checks to protect the polder framework, and visibility diagnostics remain a necessary screening layer. More broadly, the framework provides a transparent and replicable procedure that can be transferred and locally recalibrated for heritage-sensitive rural-urban fringes where change is incremental and cumulative, supporting a stage-comparable VIA approach.