From a User Study to a Valid Claim

How to Test your Hypothesis and Avoid Common Pitfalls

Conference paper (2017)

Authors

N.H.L.C. de Hoon Computer Graphics and Visualisation - EEMCS
E. Eisemann Computer Graphics and Visualisation - EEMCS
A. Vilanova Bartroli Computer Graphics and Visualisation - EEMCS
Research Group
Computer Graphics and Visualisation (EEMCS) (TU Delft)
More Info
expand_more

Published Date

2017

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Electrical Engineering, Mathematics and Computer Science

Department

Intelligent Systems

Research Group

Computer Graphics and Visualisation

Abstract

The evaluation of visualization methods or designs often relies on user studies. Apart from the difficulties involved in the design of the study itself, the existing mechanisms to obtain sound conclusions are often unclear. In this work, we review and summarize some of the common statistical techniques that can be used to validate a claim in the scenarios that are commonly present in user studies in visualization, i.e., hypothesis testing. Usually, the number of participants is small and the mean and variance of the distribution are not known. Therefore, we will focus on the techniques that are adequate within these limitations. Our aim for this paper is to clarify the goals and limitations of hypothesis testing from a user study perspective, that can be interesting for the visualization community. We provide an overview of the most common mistakes made when testing a hypothesis that can lead to erroneous claims. We also present strategies to avoid those.

Files

Eurorv3.pdf
(.pdf | 0.293 Mb)