The broad adoption of integrated computing systems through the Internet of Things has led to a need for seamless, low-latency interfaces. Deploying these installations in public spaces, such as gaming areas in airports, presents unique challenges. Traditional input modalities are ill-suited for these environments. Cameras compromise user privacy, and wearables are susceptible to theft, damage or loss. Millimetre wave (mmWave) radars are an ideal sensor for such an interface, as they can operate through non-conductive materials and independently of lighting conditions. Most human sensing solutions for mmWave radars use Deep Learning models, which rely on large data sets for training and have a high computational overhead, limiting their feasibility on resource-constrained edge devices. To address these limitations, we present IAmMuse, a lightweight, signal-processing-based interpretation pipeline for human sensing using mmWave radar technology. We filter and enhance the raw point cloud data, using spatio-temporal density information, as well as kinematic context acquired through a few-shot online learning step. IAmMuse uses this pre-processed data to generate a stabilised prediction, classifying the user’s arm position. We implemented a musical system, controlled through these predictions, as an example application for the technology. The user selects musical notes by moving their arms to either a low, middle, or high position, similar to a conductor. To assess the efficacy of this method, we present a comparative analysis with a State-of-the-Art Human Pose Estimation model. This comparison shows that IAmMuse achieves a classification accuracy 50% higher than the State-of-the-Art model, while using less than 1% of the training data. This thesis validates the viability of non-deep-learning-based interpretation algorithms for human sensing with mmWave radars through a fully functional prototype.

Modern building facades and indoor partition walls feature large amounts of transparency for sufficient lighting and social safety. However, this transparency leads to concerns about privacy invasion, as sensitive objects, such as computer monitors, are exposed to onlookers. The advent of advanced screen technology has introduced VideowindoW, a smart installation capable of adjusting the transparency of its pixels to create a self-fading window, potentially addressing these privacy concerns. This thesis investigates the combined use of these smart windows together with mmWave radar technology, as a non-intrusive method to perform human posture estimation among multiple people. It develops a real-time system that detects passersby, localizes their head/eyes and obstructs their line-of-sight to sensitive indoor content, by projecting opaque squares on the smart screen. A notable gap in the mmWave literature is the insufficient handling of posture estimation challenges posed by multiple and dynamically moving targets. To address this gap, we propose the first, to our knowledge, mmWave-based Multi-Person Pose Estimation (MPPE) system. This system combines and improves two state-of-the-art methods for tracking and posture estimation and introduces a novel dataset for dynamic targets, including ground truth data for 19 human joints. Our solution demonstrated a 20% improvement in joint localization Mean Average Error (MAE) over the baseline system, in offline experiments with a single dynamic target. Furthermore, it achieved a mean blocking accuracy of 92% in online evaluations involving multiple people and varying environment. These results highlight a promising application in privacy shielding and lay the groundwork for further research in mmWave posture estimation in more unconstrained scenarios.

Human Pose Estimation using Millimeter Wave radars has emerged as a promising alternative to traditional camera-based systems, addressing privacy and deployment constraints. While state-of-the-art Deep Learning models predominantly focus on spatial feature extraction to determine the positions of key points in the human body, this research investigates the effects of incorporating temporal dynamics in such models. It focuses of modifying an existing state-of-the-art spatial model to account for temporal dynamics and compares the performance of the two models. Long Short-Term Memory networks are used to capture temporal dependencies between frames of point clouds which significantly boosts the precision of key point detection. The proposed temporal model demonstrates a 53% reduction in Mean Absolute Error and a 45% reduction in Root Mean Squared Error compared to state-of-the-art model. Moreover, these improvements were achieved with a less complex model architecture and similar training times. The robustness of the model was further validated on a different dataset, showcasing its potential for broad application in fields such as healthcare, sports analysis, traffic monitoring and robotics. This study underscores the efficacy of temporal dynamics in pose estimation, and showcases the advantages of accounting for temporal dependencies when evaluating more complex movements.

Interactive video games often use vision-based systems or wearables to track player movements. Vision-based systems are privacy-invasive, and wearables require frequent recalibration and recharging. Frequently-Modulated Continuous-Wave (FMCW) radars have been proposed as an alternative tracking solution addressing these problems. Working in the millimeter-wave (mmWave) range, they capture scenes as point clusters, ensuring privacy without attaching sensors to the user. Previous research has shown their applicability in rehabilitation, gait recognition, and smart home appliances. This study focuses on integrating an mmWave sensing device with an interactive version of the Breakout video game. We propose a general framework with three main modules - generating points, clustering them, and reconstructing the player's movements as in-game commands. Our system enhances an already existing Kalman filter-based tracking algorithm. Online experiments were conducted to compare the proposed system to the baseline algorithm. The proposed system decreases the standard deviation on the estimated target location by 33% against motionless targets, while maintaining the baseline accuracy when tested on moving targets. Furthermore, it allows a higher game refresh rate, thus smoothing in-game movements. These results demonstrate the potential of FMCW radars in enhancing interactive video game experiences.

Crowd-control is an emerging problem in urban areas and cameras are commonly seen as the solution, however, there are major concerns regarding privacy. To overcome these issues, while still maintaining the ability to keep track of people, mmWave sensors can be utilized instead, but this introduces new challenges when it comes to people counting. They work by recording the data as point clouds, making it difficult to determine the real number of people when they are occluded. To address this challenge, we combine the spatial and temporal information of the point clouds into graphs, and explore the possibilities of Graph Neural Networks. Our system classifies up to 5 people and bikes with an accuracy of 80.47%. This accuracy is 2.53% worse than the state of the art people counting system for mmWave radar point clouds.

The modern workplace often exposes individuals to privacy risks, such as the unauthorised visibility of their computer screens. MARS (mmWave-based Assistive Rehabilitation System for Smart Healthcare), coupled with VideowindoW screens, offers an innovative solution to these threats by using mmWave radar to reconstruct human poses and estimate the position of 19 key joints. This enables the screens to become opaque based on the viewer's position, ensuring privacy. Although originally designed as a rehabilitation system, MARS can be utilised for its pose estimation capabilities to enhance workplace privacy. This research explores two modifications to the MARS architecture to assess their impact on the system's accuracy and performance. Specifically, we modify the MARS architecture by increasing the depth of its convolutional neural network (CNN) and integrating the PointNet architecture. Results establish that an optimal CNN configuration with two convolutional and two dense layers, followed by the output layer, modestly improves joint location estimation. However, integrating PointNet does not improve performance, likely due to PointNet's limitations in capturing the necessary local structural details of point clouds. These findings inform future research of possible improvements when leveraging the MARS dataset in the fields of privacy enhancement and smart healthcare applications.

The People Counting Problem requires calculating the number of people in a region of interest. This is needed in crowd-monitoring scenarios but has become increasingly problematic when relying on video cameras, as they raise privacy concerns. Instead, we propose using a mmWave radar to detect people by creating point clouds from their radar signal reflections. This approach, however, can pose challenges when people walk closely together because their individual point clouds overlap and are seen as a single, larger cloud. It is difficult to count how many individuals this large point cloud holds, which can lead to miscounting the people in the scene. One approach to address this issue is leveraging the time dimension in people walking sequences, which can be done with Long Short-Term Memory (LSTM) models. Given this, we investigate how two state-of-the-art models, PointNet and MARS, perform for people counting from point clouds when extended through LSTMs. The results show how both PointNet and MARS improve performance when extended by LSTMs. Particularly, despite having over double the parameters, MARS+LSTM outperforms PointNet+LSTM in terms of accuracy and computational efficiency. MARS+LSTM can effectively capture small changes in the local structure of point clouds between frames, which PointNet loses due to max pooling. This highlights the importance of selecting a model architecture, like the CNN in MARS, that aligns with the data characteristics to maximise performance.