“It Should Be Relevant, Reliable and Feasible”

Abstract

Face validity indicates to what extent participants are engaged in making choices; and understand and interpret the information presented to them in the study as intended by its designer. It is an important but often overlooked aspect of the overall validity of choice experiments and no comprehensive instruments for assessing it are available. Improving the design of choice experiments potentially improves the quality of participants’ responses, which increases the relevance and usability for policy and practice. In this study we developed and tested an instrument to assess the FAce validity of Choice Experiments (FACE) in a uniform, systematic manner. The instrument is based on nine components that are used to define face validity identified from literature: acceptance, clarity, completeness, familiarity, feasibility, legibility, relevance, sensitivity, and transparency. FACE covers these components in 14 questions with 5-point Likert scales on which participants can indicate their level of agreement. 1,020 participants completed the instrument following a discrete choice experiment on COVID-19 pandemic preparedness measures in the Netherlands. This first application of FACE showed that the face validity of a choice experiment was determined by whether participants considered its study design to be relevant, reliable and feasible. Moreover, we found that relevance and reliability were most strongly related to characteristics of the survey design, while feasibility was most strongly related to participants’ socio-demographic characteristics. Face validity was assessed high(er) by participants who were younger, male, lower educated, vaccinated against COVID-19, supportive of policy responses to a pandemic situation and sufficiently engaged in the experiment.