Adapting the Robust Effect Size Cliff’s Delta to Compare Behaviour Profiles

Adapting the Robust Effect Size Cliff’s Delta to Compare Behaviour Profiles

Bais, Frank (Cito Institute for Educational Measurement)
van der Neut, J.R. (TU Delft ImPhys/Medical Imaging)

2022

Cliff’s Delta is a non-parametric effect size that is based on data observations. In this paper, we elaborate on an adaptation of Cliff’s Delta in order to compare behaviour profiles. Behaviour profiles are density distributions in which survey answer behaviour is summarized for specific groups of respondents or items. Such profiles are useful, as they take into account the varying number of survey items that is filled out per respondent due to filter questions. By the adapted profile-based Cliff’s Delta, two subgroups of respondents (for instance higher and lower educated respondents) can be compared on the occurrence of specific answer behaviour (for instance giving ‘don’t know’-answers). By means of simulations, we show that the adapted profile-based Cliff’s Delta converges towards the original Cliff’s Delta as the number of items that is filled out by respondents increases. The uncertainty that comes along with a finite number of items is taken into account by the adapted profile-based Cliff’s Delta. As a result, the adapted profile-based Cliff’s Delta has a restricted magnitude for a finite number of items. We conclude that the adapted profile-based Cliff’s Delta is a solid and conservative statistic that is both useful and advantageous to compare behaviour profiles. We close with two survey data examples and by discussing our findings.

answer behaviour
behaviour profiles
Cliff’s Delta
effect size
survey statistics

http://resolver.tudelft.nl/uuid:02e9d7b2-f0eb-4355-aec9-9551685226d3

https://doi.org/10.18148/srm/2022.v16i2.7908

Survey Research Methods, 16 (3), 329-352

Institutional Repository

journal article

© 2022 Frank Bais, J.R. van der Neut