The new generation of power plants based on innovative thermodynamic cycles operating with unconventional working fluids, such as CO2 close to its thermodynamic critical point or organic fluids close to their vapour saturation line, is an attractive option for high efficiency conversion of sustainable energy sources in the low and medium power output range. One of the most challenging tasks is to make this technology competitive by realizing highly efficient turbomachinery components. The main difficulty stems from the strongly non-ideal fluid dynamic behaviour of such fluids, which largely deviates from that of fluids commonly employed in standard applications. As a consequence, special design is needed, as available methods for turbomachinery of steam or gas turbine power plants cannot be used. The scarce knowledge on both the numerical modeling of non-ideal fluid flows and on their performance in turbomachinery applications makes research crucial. Computational fluid dynamics (CFD) is a powerful tool to investigate the physics of non-ideal flows and to assist the design of unconventional machines. This thesis presents original research in the field of non-ideal fluid dynamics on three complementary aspects, namely numerical methods, applications, and theory, aiming at (i) developing new numerical schemes for the accurate and efficient CFD simulation of non-ideal compressible flows; (ii) investigating the performance of turbomachinery operating with unconventional fluids; (iii) studying the hydrodynamic stability of variable property fluid flows. The core of the report consists of five self-contained chapters, each addressing a specific problem, which are grouped into three parts according to their area of contribution. In the first part, novel implicit schemes for the spatial discretization of the Navier--Stokes equations in the framework of the finite-volume method are introduced (Chapter 2). The look-up table approach is extensively analyzed and proposed as an accurate and efficient alternative to the direct solution of the equation of state model for fluid property evaluation. In order to address the proper handling of sub-domains interfaces needed in unsteady turbomachinery simulations, a new flux-conserving treatment of non-conformal mesh blocks interfaces is proposed in Chapter 3. In the second part, the focus shifts to the investigation of non-ideal fluid flows in turbomachines. Chapter 4 reports the CFD computation of the performance map of a 20 mm diameter radial compressor operating with supercritical CO2 designed for a power of 50 kW and a rotational speed of 75 krpm. Results obtained by means of Reynolds-averaged Navier--Stokes (RANS) simulations are compared to experimental measurements of isentropic efficiency and specific enthalpy rise, showing a reasonable agreement and providing a validation for the numerical schemes introduced in Chapter 2. Chapter 5 investigates the unsteady operation of a supersonic turbine for organic Rankine cycle applications using unsteady RANS simulations. The evolution of shock waves and viscous wakes and their interaction are described in details and related to the fluctuations of blade loads, which can be of the same order of magnitude of their time-averaged values, demonstrating the primary importance of performing unsteady simulations when studying such non-conventional machines. A third part concludes the thesis presenting an analytical study of the hydrodynamic stability of a thermally-stratified channel flow with temperature dependent properties. The classical linear stability analysis framework is extended to include the effect of variable transport properties. Various distributions of thermal conductivity and specific heat across the channel height are considered in order to assess under which conditions variable properties contribute to the stabilization or destabilization of the flow. The presented results have practical implications in the study of the transition to turbulence of wall bounded flows of unconventional fluids and in devising flow control strategies.