Machine learning parameterizations of atmospheric optical turbulence for free-space optical communication

Abstract

Free-space optical communication (FSOC) is a key technology to meet the growing demand for high-bandwidth, secure, and energy-efficient data links.
However, the atmospheric channel introduces a major challenge: optical turbulence (OT). Turbulent fluctuations of the refractive index, driven by wind shear and buoyancy in the atmosphere, distort the propagating optical beam, degrading communication performance. This dissertation investigates the modelling of optical turbulence, quantified through the refractive index structure parameter C_n², along two complementary avenues: traditional numerical weather prediction (NWP) using mesoscale models and machine learning (ML) techniques.

The first part establishes the state of the art in mesoscale C_n² modelling for FSOC. Chapter 2 presents a systematic intercomparison of the three main classes of C_n² parameterizations -- flux-based, gradient-based, and variance-based -- applied to both observed data and output from a numerical mesoscale model.
Evaluated against scintillometer observations, the variance-based parameterization yields the best overall performance and, unlike the other two, is not restricted to the atmospheric surface layer. Building on this foundation, chapter 3 proposes an end-to-end framework that translates C_n² estimates into FSOC link performance metrics, specifically turbulence-induced losses and a theoretically achievable information rate (generalized mutual information, GMI).
Applied to an urban example link, the framework reveals that the sensitivity of the estimated FSOC performance to C_n² errors strongly depends on the specific link. Reporting C_n² model errors alone is therefore considered insufficient for FSOC, and end-to-end assessments of link performance are needed.

Running mesoscale models is computationally expensive, and the traditional parameterizations used to obtain C_n² -- while physically motivated -- are often limited in their applicability or accuracy. Machine learning offers a promising alternative by learning complex relationships directly from data, as explored in the second part of this dissertation. Chapter 4 introduces OTCliM, a gradient boosting-based methodology that relates globally available ERA5 reanalysis data to observed near-surface C_n². Across 17 diverse stations in New York State, OTCliM trained on just one year of observations accurately extrapolates C_n² over four held-out years. However, the resulting models are limited to near-surface C_n², which is sufficient for terrestrial links but not for satellite-to-ground applications that require vertical profiles. To extend modelling into the vertical, chapter 5 introduces Π-ML, a physics-inspired ML framework that combines automated dimensional analysis with gradient boosting. By expressing inputs and outputs as non-dimensional groups grounded in physical principles, Π-ML derives an interpretable, data-driven similarity theory for C_n² in the atmospheric surface layer. The framework achieves high accuracy under both stable and unstable atmospheric conditions and identifies scalings that are physically consistent with established theories. Finally, chapter 6 addresses the full atmospheric column by developing OTProf. This deep-learning-based model estimates C_n² profiles of high vertical resolution from low-resolution ERA5 pressure-level inputs. Trained on one year of mesoscale simulations, the model substantially outperforms the commonly used Hufnagel-Valley analytical model.
The estimated profiles exhibit a physically realistic vertical structure, although ML-typical smoothing leads to some underestimation of integrated turbulence parameters, such as the Fried parameter and the scintillation index.
Nevertheless, OTProf is regarded as a computationally efficient and considerably more accurate approach compared to traditional analytical models.

Taken together, this dissertation demonstrates that both numerical mesoscale modelling and machine learning deliver practical advances for optical turbulence estimation and FSOC applications. The overarching conclusion, however, is that the greatest potential lies in combining the two paradigms, which is viewed as a promising direction for future research.