Strategy games provide a compelling testbed for developing human-like computer agents, with applications that extend beyond gaming into fields requiring adaptive and socially intelligent AI. In these games, players tend to enjoy and engage more deeply with AI opponents that not only provide a challenge but also behave in ways that resemble human thinking and decision-making. However, despite progress in developing such agents, there is still no standard approach for evaluating how human-like these opponents truly are—making it difficult to assess and improve their design. Here I show that strategy game opponents having more human-like game-level playstyles does not necessarily lead to them being more believable (perceived as human-like by human players).

By developing a turn-based strategy game and evaluating Hierarchical Reinforcement Learning (HRL) agents of varying complexity, I assessed both their behavioural similarity to human players and how believable they were perceived to be by human players. This research introduces a new approach for understanding player behaviour using behaviour vectors composed of three high-level metrics—Aggressiveness, Management, and Exploration—consistent with existing literature. These metrics are designed to be broadly applicable across strategy games, enabling consistent comparison between human and AI opponents, as well as across different games and agents. The findings demonstrate that while HRL agents can replicate human-like playstyles without using human training data, players judge human-likeness more on perceived intelligence and fairness. This suggests that creating truly human-like AI opponents requires not just replicating human game-level playstyles, but designing agents that align with players' expectations for intelligent and fair decision-making.