The accurate computation of the irradiance incident on the surface of photovoltaic modules is crucial for the simulation of the energy yield of a photovoltaic system. Depending on the geometrical complexity of the surroundings, different approaches are commonly employed to calculate the irradiance on the photovoltaic system. In this article, we introduce a backward ray tracing simulation approach to calculate the irradiance on photovoltaic systems in geometrically complex scenarios. We explain how the repetition of time-consuming simulation steps can be avoided with the proposed approach by storing a selection of the results from the most computationally expensive parts of the problem, and we show that the irradiance calculated with the proposed approach is in good agreement with the results of Radiance, a well-established irradiance simulation tool. Furthermore, we present an experimental validation carried out using a pyranometer and a reference cell over a period of 6 months in a complex scenario, which shows errors lower than 5% in the calculation of the daily irradiation. Finally, we compare high-resolution spectral simulations with measurements taken with a spectroradiometer under different sky conditions. The proposed approach is particularly well-suited for the simulation of bifacial and tandem photovoltaic modules in complex urban environments, for it enables the efficient simulation of high-resolution spectral irradiance in scenarios with time-varying reflectance properties.