VideoMuscle

VideoMuscle: Estimating human musculoskeletal model using neural networks from monocular video

Yang, Wei-Tse (TU Delft Electrical Engineering, Mathematics and Computer Science)

van Gemert, J.C. (mentor)
Lofi, C. (graduation committee)

Delft University of Technology

2021-09-08

We present the first deep learning approach to estimate the human skeletal system of the musculoskeletal model from monocular video. The current practice of musculoskeletal modeling relies on a motion capture system and OpenSim. The data is recorded in a restricted environment, and OpenSim workflow for musculoskeletal modeling is costly. Our research aims to solve the problem by inferring the musculoskeletal model from monocular video. Thus, our study can be applied to an arbitrary environment. We propose a spatio-temporal model to regress body scales, joint angels and one rotation matrix and exploit these three components to form a musculoskeletal model. In addition, we merge the spatial transformation of the musculoskeletal model into neural networks as an end-to-end layer to measure joint positions and marker positions. Our research demonstrates that deep learning methods can directly estimate the skeletal system of the musculoskeletal model from video effectively and efficiently. Furthermore, our proposed method has more accurate predictions on body scales and joint angles than the combination of current human pose estimation methods with OpenSim.

Deep Learning
Musculoskeletal Model
Human Pose Estimation
Joint Kinematics
Convolutional Neural Networks
Transformer
Computer Vision

http://resolver.tudelft.nl/uuid:7d3bccab-168b-440b-8c80-a4f5cfdacf2b

Student theses

master thesis

© 2021 Wei-Tse Yang