The face validity of the Participatory Value Evaluation method

Abstract

This study evaluates the face validity of the Participatory Value Evaluation (PVE) method. The PVE is an online method that enables participation for a large group of participants. This method is designed for citizens to evaluate a complex situation from a policy makers point of view. If a consultation lacks face validity, there is a likelihood that the results obtained from this instrument provide false information and lead to decisions that are misleading. During this research face validity is assessed by a four-step mixed methods approach. First, a framework to assess face validity in the PVE method is drawn up and applied in a case study of the Schiphol Environmental Council. The goal of this case study is to research by means of a PVE consultation how to improve the participation and information facilities for local residents, which are the respondents, around Schiphol Airport. The framework consists out of the categories clarity, unambiguity, relevancy, readability and completeness. Second, the results of this framework are applied in an experiment. From this experiment it becomes clear that there is no difference in the evaluation of face validity by respondents between a ‘sliders’ choice task and a ‘points’ PVE. However, respondents are more indifferent towards the ‘sliders’ PVE. On average, clarity and unambiguity are evaluated the lowest and readability the highest. Third, comparing to other previous PVE case studies that includes elements of face validity, it becomes clear that the four properties impact on personal life, the respondents, the platform and who is in charge influence the differences in the evaluation of face validity between different case studies. With the three face validity categories included in previous case studies, benchmarks are established. If the goal is to score high on the clarity category, the aim should be to achieve an average evaluation score above 3,67 for clarity, above 3,88 for relevancy and around or higher than 4,11 for completeness on a five-point Likert scale. Fourth, the established framework from literature is compared with concerns about face validity in practice. In addition to the five categories, feasibility turns out to be a practical concern. It is remarkable from the results that 18- to 34-year-olds and people with a low level of education generally rate the face validity categories lower. Therefore, a recommendation for further research is to investigate how the face validity of these groups can possibly be increased and what their needs are regarding face validity.