Interactive techniques for level editing of sonification games
Q. ren (TU Delft - Electrical Engineering, Mathematics and Computer Science)
A.R. Bidarra – Mentor (TU Delft - Computer Graphics and Visualisation)
C.C.S. Liem – Graduation committee member (TU Delft - Multimedia Computing)
R. Schaefer – Graduation committee member (Universiteit Leiden)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Gesture-based sonification games offer a promising medium for motor-skill rehabilitation by transforming therapeutic movements into interactive musical experiences. However, creating custom levels for such games remains inaccessible to therapists and researchers, who often lack the musical or technical expertise required to shape musical material into meaningful motor tasks. This thesis investigates how non-technical and non-musician users can be supported in generating music, authoring interaction structures, and validating playable levels for gesture-based sonification games.
To address this challenge, this work introduces an end-to-end authoring pipeline that combines symbolic music generation with an intuitive level-editing system. A hierarchical diffusion model is adapted to produce structured, multi-track musical material through high-level controls such as key, tempo, and song form. A web-based authoring interface then allows users to refine this material, simplify dense passages through a trigger--support note mechanism, and map notes to spatially and temporally aligned gesture targets. A target-configuration module provides synchronized previews and export functions that integrate directly with the PIZZICATO runtime, enabling real-time testing and performance logging.
A qualitative expert evaluation with nine therapists and researchers examined the usability, flexibility, and therapeutic potential of the system. Participants found the workflow accessible and intuitive, valued the direct manipulation of musical and spatial elements, and highlighted the potential of the tool to streamline content creation for motor-rehabilitation studies. The evaluation also surfaced conceptual limitations, including the need for broader musical genres beyond pop-derived structures, and the opportunity to incorporate clinically informed automation such as predefined motor-exercise patterns.
This thesis contributes (i) a novel, integrated workflow for non-technical authoring of gesture-based sonification levels, (ii) interface techniques that translate symbolic musical structure into spatial--temporal interaction tasks, and (iii) empirical insights into the needs of therapists and psychologists designing movement-based therapeutic content.