Aberrant attention allocation has been implicated in the etiology and maintenance of a range of psychopathologies. However, three decades of research, relying primarily on manual response-time tasks, have been challenged on the grounds of poor reliability of its attention bias indices. Here, in a large, multisite, international study we provide reliability information for a new eye-tracking-based measure of attention allocation and its relation to psychopathology and age. Data from 1567 participants, across a wide range of psychiatric diagnoses and ages, were aggregated from nine sites around the world. Of these, 213 participants also provided retest data. Acceptable overall internal consistency and test-retest reliability were observed among adult participants (Cronbach’s alpha = 0.86 and r(213) = 0.89, respectively), as well as across all examined psychopathologies. Youth demonstrated lower internal consistency scores (Cronbach’s alpha = 0.65). Finally, the percent dwell time index derived from the task statistically differentiated between healthy participants and participants diagnosed with social anxiety disorder, major depression, and post-traumatic stress disorder. These results potentially address a long-standing reliability crisis in this research field. Aberrant attention allocation patterns in a variety of psychiatric disorders may be targeted with the hope of affecting symptoms. The attention allocation index derived from the matrix task offers reliable means to measure such cognitive target engagement in clinical contexts.