Background: The combination of genetics and imaging has improved their understanding of the brain through studies of aggregate measures obtained from high-resolution structural imaging. Voxel-wise analyses have the potential to provide more detailed information of genetic influences on the brain. Here they report a large-scale study of the heritability of gray matter at voxel resolution (1 × 1 × 1 mm). Methods: Validated voxel-based morphometry (VBM) protocols were applied to process magnetic resonance imaging data of 3,239 unrelated subjects from a population-based study and 491 subjects from two family-based studies. Genome-wide genetic data was used to estimate voxel-wise gray matter heritability of the unrelated subjects and pedigree-structure was used to estimate heritability in families. They subsequently associated two genetic variants, known to be linked with subcortical brain volume, with most heritable voxels to determine if this would enhance their association signals. Results: Voxels significantly heritable in both estimates mapped to subcortical structures, but also voxels in the language areas of the left hemisphere were found significantly heritable. When comparing regional patterns of heritability, family-based estimates were higher than population-based estimates. However, regional consistency of the heritability measures across study designs was high (Pearson's correlation coefficient = 0.73, P = 2.6 × 10⁻¹³). They further show enhancement of the association signal of two previously discovered genetic loci with subcortical volume by using only the most heritable voxels. Conclusion: Gray matter voxel-wise heritability can be reliably estimated with different methods. Combining heritability estimates from multiple studies is feasible to construct reliable heritability maps of gray matter voxels. Hum Brain Mapp 38:2408–2423, 2017.