“It should be relevant, reliable and feasible”

Abstract

Face validity indicates to what extent participants are engaged in making choices; and understand and interpret the presented information as intended by its designer. It is an important but often overlooked aspect of the overall validity of choice experiments and no comprehensive instruments for assessing it are available. Improving its design potentially improves the quality of participants’ responses and the study itself, which increases the relevance and usability for policy and practice. In this study we developed and tested an instrument to assess the FAce validity of Choice Experiments (FACE) in a uniform, systematic manner. The instrument is based on 9 components identified in the literature: clarity, completeness, decision certainty, familiarity, feasibility, legibility, relevance, sensitivity, and transparency. FACE covers these components in 13 statements with 5-point Likert scales. 1 020 participants completed the instrument following a discrete choice experiment on COVID-19 pandemic preparedness measures in the Netherlands. This first application of FACE showed that the face validity of a choice experiment was determined by whether participants considered its study design to be relevant, reliable and feasible. Moreover, we found that relevance and reliability were most strongly related to characteristics of the survey design, while feasibility was most strongly related to participants’ socio-demographic characteristics. Face validity was rated high(er) by participants who were younger, male, lower educated, vaccinated against COVID-19 and sufficiently engaged in the experiment. FACE should be regarded as a first-version instrument that can be refined and further validated. We provide recommendations on how to improve FACE in future research.