For a better understanding of how different diseases (e.g. neurovascular diseases, neurodegenerative diseases, and musculoskeletal pain conditions) affect human motor function, a uniform, standardized and objective evaluation is a desirable goal expressed within the clinical community. We explore the capabilities of an augmented reality (AR) game that uses free hand interaction to facilitate an objective assessment of the upper extremity motor dysfunction. First, the design process of the game and the system architecture are described. Second, a study about usability of the AR framework and game engagement is presented based on an experiment we conducted with five patients and ten healthy people. Lastly, a short analysis of the accuracy of the hand data when participants performed “fingers tapping” gesture is done. The results of the study show that even though users experienced the system as physically and mentally demanding, it was engaging enough to make them complete the game. The study also shows that hand data captured is accurate enough to allow a high degree (95%) of pinching gesture recognition.