Human Pose Estimation using Millimeter Wave radars has emerged as a promising alternative to traditional camera-based systems, addressing privacy and deployment constraints. While state-of-the-art Deep Learning models predominantly focus on spatial feature extraction to determine the positions of key points in the human body, this research investigates the effects of incorporating temporal dynamics in such models. It focuses of modifying an existing state-of-the-art spatial model to account for temporal dynamics and compares the performance of the two models. Long Short-Term Memory networks are used to capture temporal dependencies between frames of point clouds which significantly boosts the precision of key point detection. The proposed temporal model demonstrates a 53% reduction in Mean Absolute Error and a 45% reduction in Root Mean Squared Error compared to state-of-the-art model. Moreover, these improvements were achieved with a less complex model architecture and similar training times. The robustness of the model was further validated on a different dataset, showcasing its potential for broad application in fields such as healthcare, sports analysis, traffic monitoring and robotics. This study underscores the efficacy of temporal dynamics in pose estimation, and showcases the advantages of accounting for temporal dependencies when evaluating more complex movements.

Interactive video games often use vision-based systems or wearables to track player movements. Vision-based systems are privacy-invasive, and wearables require frequent recalibration and recharging. Frequently-Modulated Continuous-Wave (FMCW) radars have been proposed as an alternative tracking solution addressing these problems. Working in the millimeter-wave (mmWave) range, they capture scenes as point clusters, ensuring privacy without attaching sensors to the user. Previous research has shown their applicability in rehabilitation, gait recognition, and smart home appliances. This study focuses on integrating an mmWave sensing device with an interactive version of the Breakout video game. We propose a general framework with three main modules - generating points, clustering them, and reconstructing the player's movements as in-game commands. Our system enhances an already existing Kalman filter-based tracking algorithm. Online experiments were conducted to compare the proposed system to the baseline algorithm. The proposed system decreases the standard deviation on the estimated target location by 33% against motionless targets, while maintaining the baseline accuracy when tested on moving targets. Furthermore, it allows a higher game refresh rate, thus smoothing in-game movements. These results demonstrate the potential of FMCW radars in enhancing interactive video game experiences.

The People Counting Problem requires calculating the number of people in a region of interest. This is needed in crowd-monitoring scenarios but has become increasingly problematic when relying on video cameras, as they raise privacy concerns. Instead, we propose using a mmWave radar to detect people by creating point clouds from their radar signal reflections. This approach, however, can pose challenges when people walk closely together because their individual point clouds overlap and are seen as a single, larger cloud. It is difficult to count how many individuals this large point cloud holds, which can lead to miscounting the people in the scene. One approach to address this issue is leveraging the time dimension in people walking sequences, which can be done with Long Short-Term Memory (LSTM) models. Given this, we investigate how two state-of-the-art models, PointNet and MARS, perform for people counting from point clouds when extended through LSTMs. The results show how both PointNet and MARS improve performance when extended by LSTMs. Particularly, despite having over double the parameters, MARS+LSTM outperforms PointNet+LSTM in terms of accuracy and computational efficiency. MARS+LSTM can effectively capture small changes in the local structure of point clouds between frames, which PointNet loses due to max pooling. This highlights the importance of selecting a model architecture, like the CNN in MARS, that aligns with the data characteristics to maximise performance.

The modern workplace often exposes individuals to privacy risks, such as the unauthorised visibility of their computer screens. MARS (mmWave-based Assistive Rehabilitation System for Smart Healthcare), coupled with VideowindoW screens, offers an innovative solution to these threats by using mmWave radar to reconstruct human poses and estimate the position of 19 key joints. This enables the screens to become opaque based on the viewer's position, ensuring privacy. Although originally designed as a rehabilitation system, MARS can be utilised for its pose estimation capabilities to enhance workplace privacy. This research explores two modifications to the MARS architecture to assess their impact on the system's accuracy and performance. Specifically, we modify the MARS architecture by increasing the depth of its convolutional neural network (CNN) and integrating the PointNet architecture. Results establish that an optimal CNN configuration with two convolutional and two dense layers, followed by the output layer, modestly improves joint location estimation. However, integrating PointNet does not improve performance, likely due to PointNet's limitations in capturing the necessary local structural details of point clouds. These findings inform future research of possible improvements when leveraging the MARS dataset in the fields of privacy enhancement and smart healthcare applications.

In the ever-evolving field of music technology, new solutions continue to emerge that enhance musical expression and creativity. This thesis introduces WaveTune, a novel lightweight hand gesture recognition system that enables real-time control of musical composition and performance through natural hand motions. WaveTune utilizes Millimeter Wave radar technology to capture gesture data, combined with optimized deep learning techniques for real-time recognition. This provides an accessible and non-intrusive platform for gesture control that enhances privacy since no visual data is recorded. Users can dynamically select tracks and control musical parameters in real-time using expressive hand motions, integrating seamlessly with music software. A key innovation of WaveTune is the development of an optimized gesture recognition model that achieves high accuracy for real-time music interaction while minimizing complexity. This is accomplished through novel optimizations to a state-of-the-art point cloud classification architecture, resulting in an efficient and tailored model for fluid musical control. Furthermore, WaveTune promotes open-source collaboration by providing full access to code, configurations and datasets, inviting the community to build upon this system.

Recently non-image-based data capturing methods such as sensors like RF, ultrasonic or radars, Wi-Fi, Bluetooth, etc for People Counting (PC) applications have gained momentum as an alternative to camera-based systems due to the preference for privacy preservation. Among them mm-wave radars are a strong choice for data capture since they consume less power, they are robust to the extremes of weather and environment. Further, they record data in the form of a collection of points making them computationally lightweight. However, such type of data becomes a challenge when used outdoors since those environments are complex. It becomes extremely difficult to separately identify individuals from the collection of points. This makes PC outcomes inaccurate when using radar data. Therefore, in our project, we aimed to address these limitations of using mm-wave radar data. We treated groups of people as a unit, distinguishing them based on the number of people in the group. In this way, we were able to count the number of people by detecting the group size. To achieve this, we processed the raw radar output data made available by the AMS Institute into a dataset to suit our chosen deep learning model. Through a literature survey, we found that computer vision algorithms based on Deep Neural Networks (DNN), detect objects while maintaining a balance between speed and accuracy. Moreover, DNNs are capable of extracting features more effectively than traditional Machine Learning models. We selected the YOLOv8 model by Ultralytics after concluding our literature survey, as the most suitable model for our research problem. However, the selected model accepts three-channel images and videos to detect objects. Therefore, we customized the YOLOv8 model as well as formatted our radar data into a four-channel tensor input which would be acceptable by the model. We were successful in detecting groups of up to four people outdoors with a detection accuracy of 79.21%.

Crowd-control is an emerging problem in urban areas and cameras are commonly seen as the solution, however, there are major concerns regarding privacy. To overcome these issues, while still maintaining the ability to keep track of people, mmWave sensors can be utilized instead, but this introduces new challenges when it comes to people counting. They work by recording the data as point clouds, making it difficult to determine the real number of people when they are occluded. To address this challenge, we combine the spatial and temporal information of the point clouds into graphs, and explore the possibilities of Graph Neural Networks. Our system classifies up to 5 people and bikes with an accuracy of 80.47%. This accuracy is 2.53% worse than the state of the art people counting system for mmWave radar point clouds.

Modern building facades and indoor partition walls feature large amounts of transparency for sufficient lighting and social safety. However, this transparency leads to concerns about privacy invasion, as sensitive objects, such as computer monitors, are exposed to onlookers. The advent of advanced screen technology has introduced VideowindoW, a smart installation capable of adjusting the transparency of its pixels to create a self-fading window, potentially addressing these privacy concerns. This thesis investigates the combined use of these smart windows together with mmWave radar technology, as a non-intrusive method to perform human posture estimation among multiple people. It develops a real-time system that detects passersby, localizes their head/eyes and obstructs their line-of-sight to sensitive indoor content, by projecting opaque squares on the smart screen. A notable gap in the mmWave literature is the insufficient handling of posture estimation challenges posed by multiple and dynamically moving targets. To address this gap, we propose the first, to our knowledge, mmWave-based Multi-Person Pose Estimation (MPPE) system. This system combines and improves two state-of-the-art methods for tracking and posture estimation and introduces a novel dataset for dynamic targets, including ground truth data for 19 human joints. Our solution demonstrated a 20% improvement in joint localization Mean Average Error (MAE) over the baseline system, in offline experiments with a single dynamic target. Furthermore, it achieved a mean blocking accuracy of 92% in online evaluations involving multiple people and varying environment. These results highlight a promising application in privacy shielding and lay the groundwork for further research in mmWave posture estimation in more unconstrained scenarios.