We introduce CaLES, a GPU-accelerated finite-difference solver designed for large-eddy simulations (LES) of incompressible wall-bounded flows in massively parallel environments. Built upon the existing direct numerical simulation (DNS) solver CaNS, CaLES relies on low-storage, third-order Runge-Kutta schemes for temporal discretization, with the option to treat viscous terms via an implicit Crank-Nicolson scheme in one or three directions. A fast direct solver, based on eigenfunction expansions, is used to solve the discretized Poisson/Helmholtz equations. For turbulence modeling, the classical Smagorinsky model with van Driest near-wall damping and the dynamic Smagorinsky model are implemented, along with a logarithmic law wall model. GPU acceleration is achieved through OpenACC directives, following CaNS-2.3.0. Performance assessments were conducted on the Leonardo cluster at CINECA, Italy. Each node is equipped with one Intel Xeon Platinum 8358 CPU (2.60 GHz, 32 cores) and four NVIDIA A100 GPUs (64 GB HBM2e), interconnected via NVLink 3.0 (200 GB/s). The inter-node communication bandwidth is 25 GB/s, supported by a DragonFly+ network architecture with NVIDIA Mellanox InfiniBand HDR. Results indicate that the computational speed on a single GPU is equivalent to approximately 15 CPU nodes, depending on the treatment of viscous terms and the subgrid-scale model, and that the solver efficiently scales across multiple GPUs. The predictive capability of CaLES has been tested using multiple flow cases, including decaying isotropic turbulence, turbulent channel flow, and turbulent duct flow. The high computational efficiency of the solver enables grid convergence studies on extremely fine grids, pinpointing non-monotonic grid convergence for wall-modeled LES.

Exascale High Performance Computing (HPC) represents a tremendous opportunity to push the boundaries of Computational Fluid Dynamics (CFD), but despite the consolidated trend towards the use of Graphics Processing Units (GPUs), programmability is still an issue. STREAmS-2 (Bernardini et al. Comput. Phys. Commun. 285 (2023) 108644) is a compressible solver for canonical wall-bounded turbulent flows capable of harvesting the potential of NVIDIA GPUs. Here we extend the already available CUDA Fortran backend with a novel HIP backend targeting AMD GPU architectures. The main implementation strategies are discussed along with a novel Python tool that can generate the HIP and CPU code versions allowing developers to focus their attention only on the CUDA Fortran backend. Single GPU performance is analyzed focusing on NVIDIA A100 and AMD MI250x cards which are currently at the core of several HPC clusters. The gap between peak GPU performance and STREAmS-2 performance is found to be generally smaller for NVIDIA cards. Roofline analysis allows tracing this behavior to unexpectedly different computational intensities of the same kernel using the two cards. Additional single-GPU comparisons are performed to assess the impact of grid size, number of parallelized loops, thread masking and thread divergence. Parallel performance is measured on the two largest EuroHPC pre-exascale systems, LUMI (AMD GPUs) and Leonardo (NVIDIA GPUs). Strong scalability reveals more than 80% efficiency up to 16 nodes for Leonardo and up to 32 for LUMI. Weak scalability shows an impressive efficiency of over 95% up to the maximum number of nodes tested (256 for LUMI and 512 for Leonardo). This analysis shows that STREAmS-2 is the perfect candidate to fully exploit the power of current pre-exascale HPC systems in Europe, allowing users to simulate flows with over a trillion mesh points, thus reducing the gap between the Reynolds numbers achievable in high-fidelity simulations and those of real engineering applications.

The impact of intrinsic compressibility effects – changes in fluid volume due to pressure variations – on high-speed wall-bounded turbulence has often been overlooked or incorrectly attributed to mean property variations. To quantify these intrinsic compressibility effects unambiguously, we perform direct numerical simulations of compressible turbulent channel flows with nearly uniform mean properties. Our simulations reveal that intrinsic compressibility effects yield a significant upward shift in the logarithmic mean velocity profile that can be attributed to the reduction in the turbulent shear stress. This reduction stems from the weakening of the near-wall quasi-streamwise vortices. In turn, we attribute this weakening to the spontaneous opposition of sweeps and ejections from the near-wall expansions and contractions of the fluid, and provide a theoretical explanation for this mechanism. Our results also demonstrate that intrinsic compressibility effects play a crucial role in the increase in inner-scaled streamwise turbulence intensity in compressible flows, as compared with incompressible flows, which was previously regarded to be an effect of mean property variations alone.

We derive explicit formulas for the mean profiles of temperature (modeled as a passive scalar) in forced turbulent convection, as a function of the Reynolds and Prandtl numbers. The derivation leverages on the observed universality of the inner-layer thermal eddy diffusivity with respect to Reynolds and Prandtl number variations and across different flows, and on universality of the passive scalar defect in the core flow. Matching of the inner- and outer-layer expression yields a smooth compound mean temperature profile. We find excellent agreement of the analytical profile with data from direct numerical simulations of pipe and channel flows under various thermal forcing conditions, and over a wide range of Reynolds and Prandtl numbers.

We develop a theoretical framework for predicting friction and heat transfer coefficients in variable-properties forced-air convection, as typical of turbine blade cooling. To do this, we borrow concepts from high-speed wall turbulence, also featuring large temperature and density variations. Using the mean momentum balance and mean thermal balance equations we develop integral transformations that account for the effect of the variable fluid properties, and apply the inverse transformations to calculate the friction and heat transfer coefficients. The proposed theory is validated using a direct numerical simulation dataset spanning both heating and cooling conditions, and the predicted friction and heat transfer coefficients match DNS data with 1–2% accuracy.

A transformation that relates a compressible wall-bounded turbulent flow with nonuniform fluid properties to an equivalent incompressible flow with uniform fluid properties is derived and validated. The transformation accounts for both variable-property and intrinsic compressibility effects, the latter being the key improvement over the current state of the art. The importance of intrinsic compressibility effects contradicts the renowned Morkovin's hypothesis.

We present STREAmS-2.0, an updated version of the flow solver STREAmS, first introduced in Bernardini et al. (2021) [1]. STREAmS-2.0 has an object-oriented design which separates the physics equations from the specific back-end, making the code more suitable for future expansions, such as porting to novel computing architectures or implementation of additional flow physics. Similarly to the previous version, STREAmS-2.0 supports NVIDIA-GPU and CPU back-ends. Additionally, this version features improvements of the input/output data management, new energy and entropy preserving schemes for the discretization of the convective fluxes, recycling/rescaling inflow boundary condition, and a model for thermally perfect gases with variable specific heats. New version program summary: Program Title: STREAmS CPC Library link to program files: https://doi.org/10.17632/hdcgjpzr3y.2 Developer's repository link: https://github.com/STREAmS-CFD/STREAmS-2 Licensing provisions: GPLv3 Programming language: Fortran, CUDA Journal reference of previous version: M. Bernardini, D. Modesti, F. Salvadore, and S. Pirozzoli. STREAmS: a high-fidelity accelerated solver for direct numerical simulation of compressible turbulent flows. Comput. Phys. Commun. 263 (2021) 107906. Does the new version supersede the previous version?: Yes. Reasons for the new version: New code structure and release of new features. Summary of revisions: • The original solver [1] has been rewritten following an object-oriented design implemented through Fortran derived types that include variables and type bound procedures. The new software architecture has been designed to increase modularity and extensibility of the code, allowing users to add new back-ends and physics equations while maintaining the same code structure. This allows users to reuse portions of the code that are independent of the physics equations, the back-end, or both. The layer of computing procedures maintains a lean structure that can be highly optimized with respect to the implemented back-end. • Input handling is now based on the classic.ini format improving both user readability and input data management. • A family of new kinetic energy and entropy preserving schemes (KEEP) are now available and can be selected for stable, non-dissipative and accurate spatial discretization of the convective terms of the Navier–Stokes equations in smooth flow regions [2]. Concerning the shock-capturing flux, the improved low-dissipative WENO-Z scheme proposed by [3] is now available. • New inflow boundary conditions based on the recycling/rescale approach [4] have been implemented for the simulation of spatially evolving compressible turbulent boundary layers. Moreover, a new inflow condition based on the solution of the compressible Blasius equation is available to take into account the case of laminar boundary layers. • The constitutive relations have been generalized to take into account thermally perfect gases with variable specific heats, approximated with polynomial functions of the temperature that can be specified by the user [5]. • A new stretching function has been implemented to improve the distribution of grid nodes for the computation of wall-bounded turbulent flows. The formulation blends uniform near-wall spacing with uniform resolution in terms of Kolmogorov units in the outer wall layer, guaranteeing accuracy with higher computational efficiency [6]. Nature of problem: The code solves the compressible Navier–Stokes equations in Cartesian coordinates for a thermally perfect gas. The solver is designed for direct numerical simulation (DNS) of compressible supersonic turbulent boundary layers and various canonical configurations are supported, including turbulent channel flow, laminar and turbulent boundary layer and shock-wave/boundary layer interaction. Solution method: The equations are discretized using high-order finite difference approximations with hybrid low-dissipative/shock-capturing capabilities and the time advancement is performed using a Runge–Kutta scheme. References: [1] M. Bernardini, D. Modesti, F. Salvadore, S. Pirozzoli, STREAmS: A high-fidelity accelerated solver for direct numerical simulation of compressible turbulent flows, Comput. Phys. Commun. 263 (2021) 107906. [2] Y. Tamaki, Y. Kuya, S. Kawai, Comprehensive analysis of entropy conservation property of non-dissipative schemes for compressible flows: KEEP scheme redefined, J. Comput. Phys. 468 (2022) 111494. [3] R. Borges, M. Carmona, B. Costa, W. Don, An improved weighted essentially non-oscillatory scheme for hyperbolic conservation laws, J. Comput. Phys. 227 (6) (2008) 3191–3211, https://doi.org/10.1016/j.jcp.2007.11.038 [4] S. Pirozzoli, M. Bernardini, F. Grasso, Direct numerical simulation of transonic shock/boundary layer interaction under conditions of incipient separation, J. Fluid Mech. 657 (2010) 361–393. [5] B. J. McBride, M. J. Zehe, S. Gordon, NASA Glenn coefficients for calculating thermodynamic properties of individual species, NASA/TP 211556, NASA, 2002. [6] S. Pirozzoli, P. Orlandi, Natural grid stretching for DNS of wall-bounded flows, J. Comput. Phys. 439 (2021) 110408.

We carry out direct numerical simulation (DNS) of forced thermal convection in one-sided heated square ducts and compare the global heat transfer to the idealized case of uniformly heated walls, and also to the case of plane channel flow with one-sided heating. DNS results reveal a reduction of global heat transfer of about 10% at unit Prandtl number and friction Reynolds number Re_τ ≈ 2000, which is consistent with the values reported in early experiments. The global heat transfer of square ducts matches very well the one of plane channel flow with the same boundary conditions, at matched bulk Reynolds number based on the effective diameter.

We study turbulent flow in open channels with a free surface and rectangular cross-section, for various Reynolds numbers and duct aspect ratios. Direct numerical simulations are used to obtain accurate characterization of the secondary motions, which are found to be more intense than in closed ducts, and to scale with the bulk, rather than with the friction velocity. A notable feature of open-duct flows is the presence of a velocity dip, namely the peak velocity is achieved at some depth underneath the free surface. We find that the depth of the velocity peak increases with the Reynolds number, and correspondingly the flow becomes more symmetric with respect to the horizontal midplane. This is also confirmed from the change of the topology of the secondary motions, which exhibit a strong corner circulation at the free-surface/wall corners at low Reynolds number, which, however, weakens at higher. The structure of the mean velocity field is such that the log law applies with good approximation in the direction normal to the nearest wall, which allows us to explain why predictive friction formulae based on the hydraulic diameter concept are successful. Additional analysis shows that the secondary motions account for a large fraction of the frictional drag (up to %).

We carry out direct numerical simulations (DNS) of turbulent flow and heat transfer in pressure-driven plane channels, by considering cases with heating on both walls, as well as asymmetric heating limited to one of the channel walls. Friction Reynolds numbers up to are considered, and Prandtl numbers from to, the temperature field being regarded as a passive scalar. Whereas cases with symmetric heating show close similarity between the temperature and the streamwise velocity fields, with turbulent structures confined to either half of the channel, in the presence of one-sided heating the temperature field exhibits larger regions with coherent fluctuations extending beyond the channel centreline. Validity of the logarithmic law for the mean temperature is confirmed, as well as universality of the associated von Kármán constant, which we estimate to be. Deviations from the logarithmic behaviour are much clearer in cases with one-sided heating, which feature a wide outer region with parabolic mean temperature profile. The DNS data are exploited to construct a predictive formula for the heat transfer coefficient as a function of both Reynolds and Prandtl number. We find that the reduction of the thermal efficiency in the one-sided case is approximately at unit Prandtl number; however, it can become much more significant at low Prandtl number.

We carry out direct numerical simulations (DNS) of flow in a turbulent square duct by focusing on heat transfer effects, considering the case of unit Prandtl number. Reynolds numbers up to are considered that are much higher than in previous studies, and that yield clear scale separation between inner- and outer-layer dynamics. Close similarity between the behaviour of the temperature and the streamwise velocity fields is confirmed as in previous studies related to plane channels and pipes. Just like the mean velocity, the mean temperature is found to exhibit logarithmic layers as a function of the nearest wall, but with a different slope. The most important practical implication is the validity of the traditional hydraulic diameter as the correct reference length for reporting heat transfer data, as we show rigorously here. Temperature and velocity fluctuations also have similar behaviour, but apparently logarithmic growth of their inner-scaled peak variances is not observed here, unlike in canonical wall-bounded flows. Analysis of the split contributions to the heat transfer coefficient shows that mean cross-stream convection associated with secondary motions is responsible for about of the total. Finally, we use the DNS database to highlight shortcomings of traditional linear closures for the turbulent heat flux, and show that substantial modelling improvement in principle may be obtained by retaining at least the three terms in the vector polynomial integrity basis expansion.

We carry out direct numerical simulation (DNS) of flow in a turbulent square duct by focusing on heat transfer effects, considering the case of unit Prandtl number. Reynolds numbers up to Re_τ ≈ 2000 are considered which are much higher than in previous studies, and which yield clear scale separation between inner- and outer-layer dynamics. Close similarity between the behavior of the temperature and the streamwise velocity fields is confirmed as in previous studies related to plane channels and pipes. We find good agreement between the Nusselt number of square duct and circular pipe flow when the Reynolds number based on the hydraulic diameter is used, thus corroborating the common engineering practice. Popular engineering correlations for the heat transfer reveal deviations up to 5% with respect to DNS data, which are nicely fitted by a power law.

We present STREAmS, an in-house high-fidelity solver for direct numerical simulations (DNS) of canonical compressible wall-bounded flows, namely turbulent plane channel, zero-pressure gradient turbulent boundary layer and supersonic oblique shock-wave/boundary layer interaction. The solver incorporates state-of-the-art numerical algorithms, specifically designed to cope with the challenging problems associated with the solution of high-speed turbulent flows and can be used across a wide range of Mach numbers, extending from the low subsonic up to the hypersonic regime. From the computational viewpoint, STREAmS is oriented to modern HPC platforms thanks to MPI parallelization and the ability to run on multi-GPU architectures. This paper discusses the main implementation strategies, with particular reference to the CUDA paradigm, the management of a single code for traditional and multi-GPU architectures, and the optimization process to take advantage of the latest generation of NVIDIA GPUs. Performance measurements show that single-GPU optimization more than halves the computing time as compared to the baseline version. At the same time, the asynchronous patterns implemented in STREAmS for MPI communications guarantee very good parallel performance especially in the weak scaling spirit, with efficiency exceeding 97% on 1024 GPUs. For overall evaluation of STREAmS with respect to other compressible solvers, comparison with a recent GPU-enabled community solver is presented. It turns out that, although STREAmS is much more limited in terms of flow configurations that can be addressed, the advantage in terms of accuracy, computing time and memory occupation is substantial, which makes it an ideal candidate for large-scale simulations of high-Reynolds number, compressible wall-bounded turbulent flows. The solver is released open source under GPLv3 license. Program summary: Program Title: STREAmS CPC Library link to program files: https://doi.org/10.17632/hdcgjpzr3y.1 Developer's repository link: https://github.com/matteobernardini/STREAmS Code Ocean capsule: https://codeocean.com/capsule/8931507/tree/v2 Licensing provisions: GPLv3 Programming language: Fortran 90, CUDA Fortran, MPI Nature of problem: Solving the three-dimensional compressible Navier–Stokes equations for low and high Mach regimes in a Cartesian domain configured for channel, boundary layer or shock-boundary layer interaction flows. Solution method: The convective terms are discretized using a hybrid energy-conservative shock-capturing scheme in locally conservative form. Shock-capturing capabilities rely on the use of Lax–Friedrichs flux vector splitting and weighted essentially non-oscillatory (WENO) reconstruction. The system is advanced in time using a three-stage, third-order RK scheme. Two-dimensional pencil distributed MPI parallelization is implemented alongside different patterns of GPU (CUDA Fortran) accelerated routines.

We use the Reynolds-averaged Navier-Stokes (RANS) equations with a full Reynolds stress model (RSM) to study the effect of the corner angle in supersonic corner flow. RANS data are compared to reference direct numerical simulation of fully developed a square duct flow, which support predictive capability of secondary flows from Stress-ω RSM. We then carry out a parametric study by changing the corner angle in the range θ=45°-135°, focusing on the effect on the mean streamwise and secondary flow. The maximum strength of the secondary flows of about 0.015u∞ occurs for θ=90°, which is similar to what is found in fully developed square ducts. Secondary eddies have approximately unit aspect ratio, and they maintain their shape for different corner angles by translating in the direction parallel to the closest wall. As a result, the position of the vortex center can be described by a simple geometrical transformation of the wall-parallel coordinate. We find that small corner angles are responsible for locally relaminarization flow at the corner, but otherwise the mean streamwise velocity profiles transformed according to van Driest following the canonical law-of-the-wall.