Haptic technology focuses on the recreation of haptic information, i.e., a type of sensory input that uses tactile cues, forces, vibrations, or pressure to provide users with the sensation of touch, enabling users to interact physically with virtual or remote environments. One promising application of this technology lies in haptic training, where the possibility of using haptic feedback to facilitate or promote motor learning is studied. The focus of this paper lies on performance-enhancing haptic training methods, with a focus on designing a dynamic motor task. The objective of this paper is, therefore, to establish a preliminary framework that can be used to provide minimal haptic feedback while flying a quadcopter through a set of gates.
We focused on creating a preliminary framework that provides haptic feedback on the altitudinal axis of the quadcopter to the pilot using the control method Model Predictive Control (MPC). The haptic feedback is provided on the z-axis of a haptic Sigma.7 robot, which is also used as a remote controller to fly the quadcopter. The MPC implements the dynamical models of the quadcopter, and a haptic Sigma.7 robot, to determine the minimal force required to steer the Sigma.7 robot towards motor task completion. The system should provide minimal haptic force feedback within the proposed design requirements to prevent reliance on the assistance. We evaluated the effectiveness of our framework by evaluating its ability to control the quadcopter to the desired altitude setpoint under autonomous conditions using a haptic Sigma robot. Additionally, the design and performance of each of the individual building blocks of this framework, i.e. the quadcopter model, the haptic interface, and the MPC, were evaluated separately. The quadcopter, with the implementation of the onboard PID controllers, eliminating the steady-state errors and meeting the required settling times. The Sigma.7 model was sufficient within the established time horizon and range of operation, although shows limitations due to unmodelled frictional forces. The completed framework is capable of providing the Sigma.7 with the necessary input command to autonomously guide the quadcopter to its desired references in real-time, therefore completing its primary objective. Future work should explore improving the model components and integrating human elements into the predictive model.

Recent research suggests that haptic feedback—the use of physical stimuli to simulate tactile experiences—plays a crucial role in simulations in virtual reality (VR), as it can enhance immersion and facilitate motor learning. Unlike real-world objects, virtual objects lack the property of mass, necessitating its simulation through haptic devices to convey a realistic sense of weight. However, rendering weight remains a challenge, particularly for ungrounded haptic devices, which maintain a free range of motion but often face limitations such as high latency, side effects through noise, vibrations, and airflow, or the need for expensive equipment. In this work, we present LeVR, a low-cost, portable haptic proxy that simulates weight by rendering the vertical forces experienced when lifting objects—within the system’s constraints—by leveraging the force generated by an accelerated mass. The system comprises a linear rail and a capstan drive mechanism and ncorporates an impedance-based control scheme. We characterized the system’s response through step and frequency analyses. Results show that LeVR can produce a force output within a latency of 2.5 ms and render forces at frequencies ranging from 1 to 11 Hz. Furthermore, we conducted a pilot user study in which participants sorted five virtual objects by weight, ranging from 17 g to 227 g, solely based on the stimuli produced by our prototype. The results indicate that participants could generally distinguish between different stimuli, though limitations such as force instability, oscillations, and fatigue affected sorting accuracy. With our proposed system we aim to contribute to research on weight perception, to ultimately increase the effectiveness of skill acquisition and motor learning in VR.

This study investigates how deviations in avatar motion influence user motion in virtual reality (VR), specifically focusing on upper body and trunk motion in a virtual environment (VE). Previous research showed that user motion can be altered via the avatar follower effect, in which avatar deviations are followed by users. This is the first work exploring this effect for a deviation including the head. The primary objective was to explore whether these deviations could subconsciously guide user motions, potentially contributing to real-time motion sickness reduction in automated vehicles.
The experiment involved participants performing seated lateral leaning tasks where they were instructed to touch virtual goals with their heads. During some trials, the avatar unexpectedly deviated from the user's intended motion.
The results revealed that contrary to expectations, the avatar follower effect did not occur. Instead, an opposing effect was observed where participants' motions contradicted the avatar's deviations, particularly when the avatar stopped prior to the instructed goal. This effect was not influenced by the user's perspective (first or third person) or the scoring mechanism used in the game. However, individual personality traits, such as a tendency for autonomy or a focus on rewards, did affect the strength of the opposing effect.
These findings suggest that using avatar deviations to guide upper body and head motion in VR may not be effective, thus unsuited for applications such as motion sickness prevention in automated vehicles.

Stroke survivors often struggle with gait asymmetry post-therapy. Researchers are exploring Virtual Reality (VR) to address this problem with the help of visual feedback and virtual avatars. VR also makes repetitive tasks more enjoyable, improving patient compliance and outcomes. Motor adaptation, essential in rehabilitation, involves adjusting movements to new conditions. Studies have used forms of motor adaptation with visual feedback to distort participants' gait symmetry, making people walk asymmetrically. This approach is called implicit Visual Feedback Distortion (VFD), where visual feedback is manipulated without the user's awareness. This thesis explores using implicit VFD with avatars in immersive VR to address gait asymmetry. An experiment with 11 healthy participants tested implicit VFD by gradually increasing the step length of the avatar's right leg. Contrary to previous screen-based VFD studies, results showed no significant effect on gait symmetry. Additionally, no correlations were found between step symmetry and psychological states (presence, embodiment, motivation). We hypothesize that the high distortion level on the right foot, suboptimal virtual environment design, and reported neck pain contributed to these findings. Future research should explore different VFD designs, or look at multiple groups to see what conditions lead to more gait asymmetry during adaptation.

Rehabilitation following a stroke plays a crucial role in functional recovery and the regaining of independence. Advanced rehabilitation robots such as the Lokomat, C-mill, Rysen, and ZeroG have become integral tools for gait rehabilitation. The MATE (Minimally Actuated Tendon-based Exercise Environment) is a novel rehabilitation device that could offer a wider range of motion compared to the Lokomat while providing support for leg movement, unlike the C-Mill, Rysen, and ZeroG. However, initial testing on the MATE indicated that this passive tendon-based rehabilitation device lacks a proper connection to the wearer. Consequently, this study was conducted to design and evaluate an attachment model, ensuring that the pulling forces are comfortably transferred to the body and that the MATE fulfills its purpose of supporting and enhancing the natural gait. The study compared two user-centred attachment designs based on user experience, comfort, usability, acceptance, ability to follow the natural gait, and the difference in pulling forces on the legs. The findings revealed that the bandage design exerted greater control over the legs, while the cuff design provided more guidance. Additionally, it was observed that adhering to the user’s natural gait greatly influenced comfort and that walking with the MATE became easier with practice. Although the bandage design was preferred, the results suggest further iterations to enhance gait support, comfort, and usability.

Moving through immersive virtual reality (VR) is commonly achieved by physically walking in the real room or using other techniques like an omnidirectional treadmill or walk-in-place. Roomscale walking is most similar to normal walking but is limited by physical space. However, other techniques can cause user experience issues such as VR sickness, balance problems, and feeling unnatural. Newer locomotion techniques are available such as powered VR shoes, which are shoes with motorized treadmills underneath. While walking, the shoes drive the user backward and actively negate the forward velocity, reducing the needed physical space. Yet, there is little evidence of the effect of powered VR shoes on user experience, which part of this work addresses. Additionally, previous research shows mismatched VR motion (optical flow) can increase VR sickness, cognitive load, and break presence. However, full-gait locomotion studies often focus on the device, neglecting optical flow, and what is the best body part to control optical flow direction is still an open question. Therefore, we first developed a novel algorithm to convert leg-based walking to optical flow while walking on VR shoes, which may also be used for other full-gait locomotion techniques. We conducted a study with 20 participants to find which of four optical flow implementations, differing in VR motion direction, resulted in the best user experience. These direction conditions were based on body-mounted trackers: i) head orientation, ii) hip orientation, iii) standing foot velocity direction, and iv) average orientation of both feet. Head-oriented walking resulted in a significantly worse user experience compared to other conditions, with no significant differences among any other conditions. Additionally, we found no effect of optical flow on VR sickness, Mental Effort, and Presence, contrary to previous studies, but instead significant differences in Ease of Use, Input responsiveness, and Appropriateness, and indication that other user experience factors might be impacted more. Finally, we discovered that walking on VR shoes, although not completely comfortable and natural, was learnable within 10 minutes for all participants under 60 years old.

To offer engaging neurorehabilitation training to stroke patients, robotic exoskeletons have emerged as a beneficial tool to train motor tasks. However, in robotic therapy, the therapists are often limited in how they can interact with the patients. The training programs with exoskeletons are often standardized, not allowing the therapist to directly manipulate them and customize the training for specific patient needs. This paper presents a novel telerehabilitation system incorporating a therapist-in-the-loop haptic interface and immersive virtual reality (IVR), aimed at improving the interaction dynamics between patients and therapists during rehabilitation sessions. We conducted a human factors study involving 36 participants grouped in pairs, assigned as "Leaders" or "Followers," where the Leader had to guide the Follower (positioned in the exoskeleton) to a specific arm pose related to activities of daily living (ADL). This task was done under two conditions: demonstration and teleoperation. In the demonstration condition, the Follower could see the Leader displayed on a 2D screen in front of them showing them visually what was the correct pose whereas in the teleoperation condition, there was no visual aid but the Leader could guide the Follower using the haptic device. The study explored the efficiency, workload, and user experience of both modalities. Results indicated that teleoperation significantly reduced the time to complete tasks and facilitated more efficient and smoother movement patterns. While Leaders experienced reduced workload and increased motivation with the teleoperation condition, Followers reported less workload but also decreased motivation, suggesting a trade-off between task ease and engagement. In conclusion, the teleoperation system improves the ability for the therapist to interact with the exoskeleton but can disengage the patient.

Introduction: Movement disorders, such as Cerebral Palsy, prevent children from fully developing their motor coordination abilities reducing their quality of life. Trunk control is a core coordination competency but there is limited variety in sitting based therapies. This paper investigates a theorised trunk support device for children which uses a system of elastic ropes and pulleys to support and prevent them from falling whilst they perform trunk control training exercises in a seated position. The aim of this study is to understand the mechanical properties required to ensure this device behaves safely and predictably. The device is considered safe if it prevents the child from falling and predictable if it always pulls the child towards an upright seating position. Additionally, it was investigated if the mechanical properties could be satisfied by off-the-shelf components and two possible pulley systems were compared.

Methods: To theoretically test if this device concept could meet the three criteria, a MATLAB simulation was written. A genetic optimization was then used to find the least stringent mechanical properties that still satisfy the three criteria. Off-the-shelf components that satisfy the mechanical properties were then found.

Results: Both pulley systems satisfied the safety and predictability criteria. Moreover, the necessary mechanical properties could be satisfied by off-the-shelf components, though some design modifications may be required. Ultimately, the configuration involving shorter ropes was determined to be the better option as it had less stringent mechanical requirements and potential design modifications would likely be easier to implement.

Conclusion: This study concluded that the optimised device, theoretically, shows good performance and seems practical to build. Future work is required to design a harness that would allow a child to comfortably train with the device.

Lower-limb exoskeletons often use trajectory-tracking control to define the device's motion and assistance level. One challenge lies in ensuring a smooth and comfortable interaction between the user and the robotic device by defining a reference trajectory. While recent research has focused on generating individualized gait patterns based on user-specific body characteristics and walking speed, limited research has explored the subjective perception of these patterns and their impact on user experience and rehabilitation outcomes.

This study investigates user perceptions of individualized versus standard and random gait patterns, focusing on enjoyment, comfort, and naturalness. A predictive gait pattern model, incorporating individual data and walking speed, was developed and tested with human participants using a grounded robotic lower limb device. Participants compared the three gait pattern types and provided subjective feedback through a questionnaire.

Findings indicate no significant preference for any gait pattern in terms of enjoyment, comfort, and naturalness, except for physical strain where the predicted pattern caused significantly more strain than the standard. The analysis also revealed that longer engagement with the device led to increased comfort and naturalness, suggesting an adaptation effect. A general tendency towards preferring the standard pattern was noted, though further research is necessary to determine whether a larger sample size reveals significant differences. Additionally, the perception of different gait patterns and their effect on the rehabilitation outcome should be explored with stroke patients.

Immersive virtual reality (IVR) with head-mounted displays (HMDs) is expanding in various fields like training, but its effects on cognitive load from visual, auditory, and mental stimuli in virtual environments remain uncertain. This is particularly relevant in neurorehabilitation, where patients may suffer from training in overstimulating environments due to cognitive impairments. This study further explores how low and high levels of visual, auditory, and cognitive demands affect the cognitive load. Twenty-two participants used an HMD for a virtual shopping task, which involved selecting listed products and placing them in a cart, under baseline (the task without additional demands) and two stimulus complexity levels (low and high) for visual, auditory, and mental demands. The study assessed cognitive load using conventional (heart rate, variability, skin conductance, performance, self-reported questionnaire) and underexplored measures (head stillness, hand smoothness, gaze behavior), and explored behavior changes due to stimulus impact. Results showed that visual and auditory stimuli had minimal effects on cognitive load, with only specific measures showing any notable differences from the baseline. Mental stimuli significantly impacted cognitive load, with high mental tasks notably affecting the measures and behavior, whereas low mental tasks showed fewer changes. This research concluded that mental stimuli significantly increased the cognitive load in virtual environments, more than visual or auditory stimuli, suggesting future virtual reality designs should prioritize managing mental load. Furthermore, the study highlights the effectiveness of head stillness and gaze behavior as innovative measures for evaluating cognitive load.

Robotic rehabilitation systems that provide proprioceptive information offer a promising approach to helping stroke survivors regain lost proprioceptive functions. One potential way to provide proprioceptive information is viscosity rendering. However, how the modulated viscosity rendering affects brain activity remains unclear. To investigate the correlation between viscosity rendering and brain activity and to provide a neurological basis for the design of robotic rehabilitation systems, an experimental setup was developed to deliver various viscosity rendering and fixed stiffness rendering during hand movement. In the experiment, twelve healthy participants interacted with virtual bottles with the same bottle stiffness but filled with liquids of different viscosities under the same movement speed, providing different levels of proprioceptive information in the form of force on muscles and joints. Control conditions without viscosity and stiffness rendering were also tested, involving both passive and active hand movements at the same speed. Results showed that stronger mu-ERD and beta-ERD were observed during movements with viscosity and stiffness rendering compared to control conditions. No significant evidence suggested that different viscosity in rendering caused variations in mu-ERD or beta-ERD. Additionally, no significant differences were found between active movement without haptic rendering and passive movement in EEG activities. These findings suggest that while the existence of viscosity and stiffness rendering during movement strengthens brain activity, modulating viscosity rendering does not significantly affect this response. This insight is particularly valuable for designing robotic rehabilitation systems that incorporate viscosity rendering.

While the presence and demand of robotic rehabilitation devices are rising, not many studies have been performed on haptic communication/ telerehabilitation with these devices. Despite therapists' desire to have the possibility to remain in the loop when their patients are performing exercises. This study aims to provide a telerehabilitation system that accommodates therapist-patient interaction during robotic rehabilitation and evaluate the system empirically on stability and transparency.
The system provides a platform for further research on telerehabilitation and haptic communication in robotic rehabilitation. Working towards a system that allows the therapist, when desired, to intuitively interact with the patient while they are performing rehabilitation exercises using a robotic device. A modular telerehabilitation system is designed using the ARMin V (ETH Zurich) upper extremity exoskeleton as the patient side, and the haptic end-effector device Sigma.7 (Force Dimensions) as the therapist's side. A visualization is provided to the therapist side using Unity and additional features are added to improve usability. The telerehabilitation system is bilateral impedance controlled through a proportional-derivative controller.
An experiment is performed in which the observing participant is asked to resist motion (analyze stability) or to be compliant with the motion (analyze transparency).
The empirical analysis showed promising first results on position tracking, effective communication of haptic cues, stability, and transparency.
However, UDP communication rate could be raised, and the scaling of force and workspace between Sigma.7 and ARMin V could be better matched to improve transparency.

Robots can aid in post-stroke motor function recovery and motor learning through the use of haptic feedback during collaborative training. A clear objective in robotic-assisted motor learning is to adapt the haptic feedback to individual users, but personal characteristics are not yet considered in this adaptation.

We investigated the suitability of a haptic guidance feedback strategy, based on participants’ locus of control character trait, compared to training without haptic guidance. For this purpose, a motor learning experiment was conducted on 42 healthy participants, where the internal dynamics of a pendulum had to be learned in order to hit upcoming targets. For two groups, training either with or without haptic guidance, we assessed motor learning and its generalization to similar tasks through target hitting performance, as well as behavior during training and perceived user experience.

Evidence was found of a relatively better performance improvement in both training and long-term (generalization of) motor learning for participants with a more internal compared to external locus of control. Lower observed interaction force during training and increasingly better performance throughout training
in these participants may have caused these motor learning differences. More positive user experience in these individuals through a higher perceived control over the pendulum and lower perceived frustration with haptic guidance may have also contributed.

Combined, this suggests an intrinsically better compatibility
with haptic guidance for people with a more internal rather than
external locus of control, for motor learning, during training and
in user experience

People that are recovering from a stroke need frequent and high-intensity training to regain their upper-limb capacity. This will eventually result in better performances and higher quality of their daily life. Robotic devices are often used to assist stroke patients in rehabilitation. The minimally-supervised portable hand trainer, which is currently developed at the Motor Learning and Neurorehabilitation Lab of the TU Delft, specifically focuses on hand and forearm rehabilitation. Haptic feedback is used in a virtual game to train patients' finger flexion and extension and forearm pronosupination.
Prior to this master thesis, a feasibility and usability study has been performed with the second iteration of the hand trainer in collaboration with therapists from Rijndam and healthy participants, showing room for improvement.

In this graduation project, the portable hand trainer is redesigned to develop an improved product design in comparison to the second iteration. Following a human-centered design approach, a modified version of the double diamond method was created to achieve a redesign that enables more functional, ergonomic, and motivating rehabilitation training.

Analysis on strokes, on rehabilitation, on the state-of-the-art, through peer tests, and on the usability study, were performed to create a decision matrix to define a list of potential improvements, distinguishing high-priority, medium-priority and low-priority improvements.
The high-priority improvements have been thoroughly addressed in multiple cycles of ideation-, developing- and validating activities. The results are combined and transformed into a hardware-ready 3D prototype that demonstrates improvement in the pronosupination movement, the donning and doffing of the device and the wrist support. The prototype of the third iteration of the portable hand trainer is evaluated through interviews with experts in the field to reflect on the project goal and propose recommendations on future work.
The medium-priority improvements have been addressed in one cycle of ideation activities of which the results are presented in sketches to advice on further development.
The low-priority improvements have not been addressed in this project.

Virtual training environments are powerful tools for various applications. To increase the immersion and realism of these environments, several devices have been developed capable of rendering a wide range of haptic sensations. As humans mainly interact with objects using their hands, the main focus of research has been on developing devices capable of providing feedback to the hand and fingers. Currently, most of these devices only provide one form of feedback, either kinesthetic feedback to finger posture or tactile sensations in the form of pressing or skin stretching. These devices seldom provide more than one kind of feedback while remaining compact enough to be wearable with a natural range of motion of the hand. In this work, we present a device that combines an existing kinesthetic feedback glove with a novel tactile fingertip display. The haptic display consists of a cable-driven platform, powered by a stand-alone actuation module. It has been designed to be compact enough for a near-natural range of motion, with the possibility of multi-finger applications in mind. To validate the design, we conducted an experiment with 16 healthy young par- ticipants who were asked to reproduce virtual shapes with their index fingers after exploring those shapes under two different conditions: 1) with purely kinesthetic feedback, and 2) with kinesthetic feedback and tactile feedback. The outcome metrics were the exploration time, reproduction time, and reproduction error. For each testing condition, the workload and motivation of the participants were also evaluated. We found that participants had a lower reproduction error and used more time to reproduce the shapes with the novel haptic display enabled compared to purely kinesthetic feedback. We did not find significant differences in exploration time, workload, or motivation between testing conditions. Thus, the combined feedback provided by our novel device leads to better performance in shape reproduction compared to only kinesthetic feedback. Further, our device is lightweight and compact, potentially enabling multi-finger use which may lead to even greater performance and immersion in virtual object manipulation tasks.

The provision of somatosensory information may play a fundamental role in the recovery of stroke patients. Particularly, tactile information is known to influence motor control by contributing to the perception of the weight, friction, and slip condition of objects. Despite this, the inclusion of tactile information through haptic rendering in robotic neurorehabilitation systems remains largely unexplored. In this study, we present a tactile interface to extend the kinesthetic rendering capabilities of an existing hand rehabilitation robot. The developed solution relies on skin stretch stimulation of the fingerpads, which allows the rendering of interaction forces with tangible virtual objects, e.g., friction, weight, and inertia. In contrast with previous skin stretch devices, our system uses closed-loop force control for accurate force rendering, relying on a custom magnetic field-based three-axis force sensor. A three-axis positioning stage in combination with a reference force sensor was used for calibrating and characterizing the sensor, as well as evaluating the interface response. The sensor achieves shear force accuracies of 0.2 N, influenced by hysteresis and viscoelastic creep effects. The tactile interface achieves a steady-state error of 0.2–0.4N and rise times of 20–70 ms during step response tests. Frequency response tests show that the interface can successfully track signals up to 5–7 Hz. The novel use of force-controlled skin stretch stimulation aims to open a new avenue for the accurate rendering of interaction forces through the tactile sense. Moreover, through purposeful design for usage in the rehabilitation domain, we hope that this study will serve as a stepping stone toward the inclusion of tactile information in robot-assisted therapies.

Bicycle safety is quickly becoming an increasingly important field as the number of electric bicycles on the streets grows faster each year. E-bikes are able to accelerate quicker and travel at faster speeds than conventional bicycles, increasing the severity of injuries in case of an accident. There are a number of ways to improve safety, such as building better infrastructure, implementing active safety systems, improving bicycling skills, or better protective gear. In this thesis, a controller based on Model Predictive Control has been designed to explore whether it could be used as a haptic guidance system that would improve motor learning of a cycling task, and whether it could assist the cyclist during cycling manoeuvres. The cycling task of lane change manoeuvres performed at a constant forward velocity was investigated. The controller was implemented on a desktop PC, which wirelessly controlled the TU Delft's Steer-by-Wire bicycle -- a bicycle where the steering is enabled by the use of electric motors instead of mechanical coupling between the front fork and the handlebars. To test the controller, ten participants took part in a pilot study. The study was designed following a counterbalanced measures design and the participants were split into two groups that experienced the controller's haptic guidance in different order. During the study, the participants were asked to ride the bicycle on a treadmill and hit virtual targets, shown on a display mounted in front of the treadmill, by carrying out lane change manoeuvres. The hypothesis stated that the controller improves performance while it is assisting the participants. It was found that the controller did not significantly improve immediate performance and this result is likely caused by too low of the task difficulty. However, it is likely that the controller was more effective at improving motor learning of lower skilled participants compared to higher skilled participants, but a small number of lower skilled participants limited the analysis. A short post-hoc no-hands riding test of the same cycling task was carried out to investigate whether the controller is able to carry out lane change manoeuvres with minimal rider input. A significant performance improvement was found during the no-hands test. In conclusion, due to limitations of the study, no performance or motor learning improvement caused by the controller was found. Yet the controller showed promising results in a no-hands riding test, which suggests that the controller could be used as a starting point for advanced safety systems for bicycles.

In order to improve skill acquisition and neurorehabilitation, we need to improve our understanding of human motor learning. It has been shown that innate variability of movements made by an individual when performing a motor task (motor variability) might enhance skill acquisition. Augmenting motor variability could therefore be a promising method to enhance learning. However, current methods that enhance motor variability show divergent results and need to be better understood.
In a lab-based experiment with twenty healthy participants,
we studied the effect of a new method that haptically increases participants’ motor variability in learning a dynamic task, i.e., controlling a pendulum. This new method consisted of applying pseudo-random perturbation forces to the internal degree of freedom of the dynamic system (indirect haptic noise), instead of applying forces directly on the trainee’s hands as previously studied. The main task consisted of swinging a virtual pendulum to hit incoming targets with the pendulum ball. To assess generalization of learning we used two transfer tasks, which consisted of altered target positions or altered task dynamics (i.e., a pendulum with shorter rod length). We evaluated the effect of the new method
on learning by comparing performance gains after training to a control group who trained without perturbations. We found that the perturbations successfully increased participants’ motor variability during training. Although we observed no learning benefits of training with this indirect haptic noise for the trained
task compared to the control group, we observed divergent effects for transfer of learning. Participants that trained with indirect haptic noise seemed to benefit in transfer of learning to altered task dynamics but not in the task with altered target positions. Increasing motor variability by indirect haptic noise is promising for enhancing skill acquisition, specially in transfer of learning, and in tasks that incorporate complex dynamics. However, more research is needed to make indirect haptic noise a valuable tool for real life motor learning situations, e.g., in
robotic neurorehabilitation.

Background: The procedure to fit a prosthetic socket to a patient, which can assure the patient’s comfort during activities of daily living, is labour intensive. Such a lengthy procedure could benefit from an automated and more efficient data-driven method capable of automatically tracking the relative movement between the patient’s tibia and the prosthetic socket. To investigate such a method, we acquired in-socket bone displacement data during the physical activities of the prosthetic user. Manually tracking the location of the tibia from, e.g., B-mode (imaging) ultrasound (US) sequences might be a solution, but this is time-consuming, and the interpretation of the sequences is highly operator dependent. Therefore, an automated and efficient method to assess socket fit in US sequences is needed.
Methods: We used an existing 3D U-Net with a long short-term memory module (LSTM) and compared its ability to track a landmark location point on the tibia in US recordings by comparing the displacement and similarity in shape with data obtained from a semi-automatic single-point tracker. To evaluate the performance of the developed automated workflow, we obtained experimental data from three participants who performed three repetitive stepping tasks with their prosthetic leg in a sideways, forward, and backward motion. Three deep learning models were trained with a varying hold-out method (66% training data, 34 % test data) to test the ability to track a landmark location on the tibia in unseen data from one participant. To find the similarity of the deep learning models compared to a semi-automated single point tracker, the normalised root mean squared error (NRMSE) was calculated. We also evaluated the normalised maximum cross-correlation (NMCC) to account for the maximum similarity in displacement trajectory when a delay occurred between the true trajectory and that from the automated model. We analysed the repeatability of each step task per participant with the standard deviation from the mean tibia’s landmark location trajectories.
Results: Due to the delay between the semi-automated single-point tracker and the DL model, the NRMSEs ranged between 27% and 90%. The similarity threshold (0,95) was reached for five trajectories of the tracked point on the tibia in the anterior-posterior direction, with a delay between 1,5% and 8,5% of the step duration. The similarity in the anterior-posterior direction of the tibia’s landmark location trajectory was higher than that in the lateral-medial direction. The SD for all participants was around 1 mm but varied proportionally to the amount of movement observed per participant. The SD of the DL models was similar to that of the semi-automated single-point tracker.
Conclusion: We conclude that a DL model from a 3D U-Net with an LSTM module has the potential to assist prosthetists and researchers in tracking in-socket tibial bone movement in the anterior-posterior direction.

Neurological disorders in the nervous and neuromuscular systems affect approximately 260 million people annually and among these 255 million would benefit from rehabilitation [4]. Patients with neurological disorders usually require multi-dimensional rehabilitation, involving physical, cognitive, psychological, and medical help. Children with trunk control problems arising due to some of these neurological disorders also require such multi-dimensional rehabilitation. A major part of this is administered to the patient through the activities of a physiotherapist in the clinical context. But the limited number of physiotherapists result in exercises often being prescribed for patients as in-home rehabilitation. During in-home rehabilitation, the patient and the primary care-giver may not be able to comply with the prescription without feedback from a physiotherapist.
To address this challenge, this paper proposes an automated method for assessing movement quality of children during trunk control rehabilitation exercises. We adopted a Human-centered AI approach to the development of our system. We identified the needs of physiotherapists for assessing patient’s functional abilities through semi-structured interviews with six physiotherapists. As a result, we co-designed and developed an artificially Intelligent decision support system that automatically assesses the quality of motion. We created a trunk-control rehabilitation exercise movement dataset based on a protocol co-designed by the authors and the physiotherapists. The data was collected
from 15 typically developing children (mean age 7 years, range 4–10 years) using a ZED-mini stereo-camera and the quality scores as ground-truth were obtained from a physiotherapist. The exercises involved reaching targets kept on a table in front
while being seated away from the table on a stool. We investigated the performance of Random Convolutional Kernel transform and XCM, two state-of-the-art multivariate
time-series classification algorithms on this dataset and achieved a quality prediction f1-score of 65% on the test dataset and similar promising results on the detection of compensatory movements in the exercise motion. In addition, to increase the trust-worthiness of our AI solution, we have provided explanations on the predictions of the black-box algorithms, which can aid the users of the system to understand the causal relationships between the input and output to the AI algorithm.