Online IC parameter adjustment of an active knee prosthesis using Reinforcement Learning with frequency-domain state representations

Abstract

Active knee prostheses are potent in assisting users, providing symmetry in walking, reducing metabolic costs, and preventing long-term health problems. The heart of their complex control algorithm employs the Impedance Control (IC) Law, which controls the torque output of the device by three parameters: stiffness coefficient, equilibrium angle, and damping coefficient. Ideally, these parameters should be personalized and adaptive to address interpersonal and intrapersonal variations on level-ground walking. However, current practices achieve personalization only through basic normalization and do not address adaptiveness. This thesis aims to utilize an RL framework with a novel frequency-domain state representation to address personalization and adaptiveness simultaneously. A policy iteration algorithm from the Q-learning family was chosen as the essence of the RL framework, and bellman error (BE) was chosen as the primary evaluation metric. The study revolves around two hypotheses. First, does the RL framework is suitable for the system? Second, does the frequency-domain state representation perform better than the time-domain state representation? Within the scope of the thesis, a custom environment was created by modifying the Humanoid-v04 environment of OpenAI Gym using MuJoCo (Multi-Joint dynamics with Contact). This environment is used to train the RL framework and conduct the experiments. Results suggest that the proposed RL framework can improve the system, and the frequency-domain state representation is superior to its time-domain counterpart. The latter conclusion has an impact beyond the active knee prosthesis domain and can inspire any trajectory following tasks with periodic signals.