The characterization of optical anisotropy in thin van der Waals (vdW) materials is crucial for both fundamental studies and nanophotonic applications. However, conventional techniques such as spectroscopic ellipsometry face significant limitations in measuring out-of-plane anisotropy and require large-area, uniform films. In this work, we present a novel framework based on coherent Fourier scatterometry (CFS) combined with deep learning for the rapid, label-free characterization of in-plane and out-of-plane refractive indices of anisotropic thin films. We designed a specialized deep neural network, AnisoVision, and trained it on simulated far-field angular spectra from multilayer stacks using the 4 × 4 Berreman matrix formalism. To efficiently capture the directional dependence of anisotropy, we utilize radially polarized light and extract only three far-field azimuthal cross sections (0, 45, 90°), enabling robust retrieval while minimizing data requirements. Our method demonstrates accurate index retrieval for both isotropic and anisotropic materials, including uniaxial h-BN and biaxial α-MoO3 flakes of varying thickness. We further validate the model’s stability by testing multiple flakes of the same material across a range of thicknesses, yielding consistent optical constants. Our approach is single-shot, nondestructive, and applicable to localized sample regions, making it suitable for heterogeneous or exfoliated samples. Additionally, the technique can be readily extended to broadband operation for spectroscopic analysis. Our work establishes CFS coupled with deep learning as a powerful platform for high-throughput optical metrology of low-dimensional materials.