Selective sustained attention, or the ability to allocate perceptual and mental resources to a single object or event, is an important cognitive ability widely assumed to be required for learning. Assessing young children's selective sustained attention is challenging due to the limited number of sensitive and developmentally appropriate performance-based measures. Furthermore, administration of existing assessments is difficult, as children's engagement with such tasks wanes quickly. One potential solution is to provide assessments within an engaging environment, such as a video game. This chapter reports the design and psychometric validation of a video game (Monster Mischief) designed to assess selective sustained attention in preschool children. In a randomized controlled trial, the authors demonstrate that Monster Mischief is significantly correlated with an existing measure of selective sustained attention (rs ≥ 0.52), and more motivating for young children as almost three times more children preferred Monster Mischief to the existing measure.@en

Many game designers aim to optimize difficulty to make games that are "not too hard, not too easy." However, recent experiments have shown that even moderate difficulty can reduce player engagement. The present work investigates other design factors that may account for the purported benefits of difficulty, such as choice, novelty and suspense. These factors were manipulated in three design experiments involving over 20,000 play sessions of an online educational game. The first experiment (n=10,472) randomly assigned some players to a particular level of difficulty but allowed other players to freely choose their difficulty. Moderately difficult levels were most motivating when self-selected; yet, when difficulty was blindly assigned, the easiest games were most motivating. The second experiment (n=5,065) randomly assigned players to differing degrees of novelty. Moderate novelty was optimal, while too much or too little novelty reduced intrinsic motivation. A final experiment (n=6,511) investigated the role of suspense in "close games", where it was found to be beneficial. If difficulty decreases motivation while novelty and suspense increase it, then an implication for educational game designers is to make easy, interesting games that are "not too hard, not too boring".@en

"Multi-armed bandits" offer a new paradigm for the AIassisted design of user interfaces. To help designers understand the potential, we present the results of two experimental comparisons between bandit algorithms and random assignment. Our studies are intended to show designers how bandits algorithms are able to rapidly explore an experimental design space and automatically select the optimal design configuration. Our present focus is on the optimization of a game design space. The results of our experiments show that bandits can make data-driven design more efficient and accessible to interface designers, but that human participation is essential to ensure that AI systems optimize for the right metric. Based on our results, we introduce several design lessons that help keep human design judgment in the loop. We also consider the future of human-technology teamwork in AI-assisted design and scientific inquiry. Finally, as bandits deploy fewer lowperforming conditions than typical experiments, we discuss ethical implications for bandits in large-scale experiments in education.@en

Selective sustained attention, or the ability to allocate perceptual and mental resources to a single object or event, is an important cognitive ability widely assumed to be required for learning. Assessing young children's selective sustained attention is challenging due to the limited number of sensitive and developmentally appropriate performance-based measures. Furthermore, administration of existing assessments is difficult, as children's engagement with such tasks wanes quickly. One potential solution is to provide assessments within an engaging environment, such as a video game. This paper reports the design and psychometric validation of a video game (Monster Mischief) designed to assess selective sustained attention in preschool children. In a randomized controlled trial, the authors demonstrate that Monster Mischief is significantly correlated with an existing measure of selective sustained attention (rs ≥ 0.52), and more motivating for young children as almost 3 times more children preferred Monster Mischief to the existing measure.@en

Online games can serve as research instruments to explore the effects of game design elements on motivation and learning. In our research, we manipulated the design of an online math game to investigate the effect of challenge on player motivation and learning. To test the "Inverted-U Hypothesis", which predicts that maximum game engagement will occur with moderate challenge, we produced two large-scale (10K and 70K subjects), multifactor (2x3 and 2x9x8x4x25) online experiments. We found that, in almost all cases, subjects were more engaged and played longer when the game was easier, which seems to contradict the generality of the Inverted-U Hypothesis. Troublingly, we also found that the most engaging design conditions produced the slowest rates of learning. Based on our findings, we describe several design implications that may increase challenge-seeking in games, such as providing feedforward about the anticipated degree of challenge.

@en

Using the online educational game Battleship Numberline, we have collected over 8 million number line estimates from hundreds of thousands of players. Using random assignment, we evaluate the effects of various adaptive sequencing algorithms on player engagement and learning.@en

Traditional experimental paradigms have focused on executing experiments in a lab setting and eventually moving successful findings to larger experiments in the field. However, data from field experiments can also be used to inform new lab experiments. Now, with the advent of large student populations using internet-based learning software, online experiments can serve as a third setting for experimental data collection. In this paper, we introduce the Super Experiment Framework (SEF), which describes how internet-scale experiments can inform and be informed by classroom and lab experiments. We apply the framework to a research project implementing learning games for mathematics that is collecting hundreds of thousands of data trials weekly. We show that the framework allows findings from the lab-scale, classroom-scale and internet-scale experiments to inform each other in a rapid complementary feedback loop.@en