K.G. Langendoen
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
65 records found
1
The rapid growth of the offshore wind energy sector demands the deployment of increasingly large wind turbines, requiring specialized heavy-lift vessels equipped with massive rotary cranes. Safe and efficient load transfer during maritime operations is severely challenged by lower-block (hook) sway, induced by vessel and crane motions. Automated anti-sway control can mitigate these operational risks, but relies on accurate real-time lower-block position and velocity feedback. While state-of-the-art lower-block localization approaches typically assume a rigid crane structure or require hook-mounted active hardware, this thesis presents a real-time state estimation framework based on vessel- and boom-mounted sensors. At the massive scale of offshore cranes, structural boom elasticity introduces significant bending deflections that corrupt joint-encoder measurements. To account for these dynamics, a multi-body non-linear model incorporating a flexible boom alongside wave-induced vessel motion is derived. This model serves as the predictor step for a multi-rate filtering architecture that fuses boom-mounted 3D LiDAR data, encoders, and a vessel Motion Reference Unit.
Both Extended (EKF) and Unscented (UKF) Kalman Filters are implemented and evaluated. By explicitly accounting for structural flexibility, the framework successfully limits tracking errors, achieving a position RMSE of 1.67-3.58 cm and a velocity RMSE of 0.66-5.52 cm/s, comfortably satisfying the requirements of 5.0-10.0 cm and 5.0-10.0 cm/s per axis, respectively. The EKF systematically outperforms the UKF, yielding an average 5.2% lower position RMSE and an 11% reduction in computational runtime, demonstrating the framework's viability for real-time industrial anti-sway control. ...
Both Extended (EKF) and Unscented (UKF) Kalman Filters are implemented and evaluated. By explicitly accounting for structural flexibility, the framework successfully limits tracking errors, achieving a position RMSE of 1.67-3.58 cm and a velocity RMSE of 0.66-5.52 cm/s, comfortably satisfying the requirements of 5.0-10.0 cm and 5.0-10.0 cm/s per axis, respectively. The EKF systematically outperforms the UKF, yielding an average 5.2% lower position RMSE and an 11% reduction in computational runtime, demonstrating the framework's viability for real-time industrial anti-sway control. ...
The rapid growth of the offshore wind energy sector demands the deployment of increasingly large wind turbines, requiring specialized heavy-lift vessels equipped with massive rotary cranes. Safe and efficient load transfer during maritime operations is severely challenged by lower-block (hook) sway, induced by vessel and crane motions. Automated anti-sway control can mitigate these operational risks, but relies on accurate real-time lower-block position and velocity feedback. While state-of-the-art lower-block localization approaches typically assume a rigid crane structure or require hook-mounted active hardware, this thesis presents a real-time state estimation framework based on vessel- and boom-mounted sensors. At the massive scale of offshore cranes, structural boom elasticity introduces significant bending deflections that corrupt joint-encoder measurements. To account for these dynamics, a multi-body non-linear model incorporating a flexible boom alongside wave-induced vessel motion is derived. This model serves as the predictor step for a multi-rate filtering architecture that fuses boom-mounted 3D LiDAR data, encoders, and a vessel Motion Reference Unit.
Both Extended (EKF) and Unscented (UKF) Kalman Filters are implemented and evaluated. By explicitly accounting for structural flexibility, the framework successfully limits tracking errors, achieving a position RMSE of 1.67-3.58 cm and a velocity RMSE of 0.66-5.52 cm/s, comfortably satisfying the requirements of 5.0-10.0 cm and 5.0-10.0 cm/s per axis, respectively. The EKF systematically outperforms the UKF, yielding an average 5.2% lower position RMSE and an 11% reduction in computational runtime, demonstrating the framework's viability for real-time industrial anti-sway control.
Both Extended (EKF) and Unscented (UKF) Kalman Filters are implemented and evaluated. By explicitly accounting for structural flexibility, the framework successfully limits tracking errors, achieving a position RMSE of 1.67-3.58 cm and a velocity RMSE of 0.66-5.52 cm/s, comfortably satisfying the requirements of 5.0-10.0 cm and 5.0-10.0 cm/s per axis, respectively. The EKF systematically outperforms the UKF, yielding an average 5.2% lower position RMSE and an 11% reduction in computational runtime, demonstrating the framework's viability for real-time industrial anti-sway control.
Rust's ownership-based type system provides strong memory-safety and aliasing guarantees, but unsafe Rust allows raw pointers to alias with references, which the static borrow checker cannot track. Dynamic aliasing models such as Stacked Borrows and Tree Borrows address this gap in the Rust interpreter Miri by defining when pointer uses create undefined behaviour. However, these models are implemented directly inside Miri, so changing or comparing model variants requires modifying the interpreter and risks semantic drift between model definitions and implementations. We present BorrowMIR, a model-parametric framework for specifying Rust aliasing models as explicit state-transition rules over MIR-level borrow events. BorrowMIR represents borrow state using per-location directed acyclic graphs with allocation-level metadata, allowing stack-shaped, tree-shaped, and more general alias derivation structures. From a single model specification, BorrowMIR generates a Rust backend for execution inside Miri and a TLA+ backend for trace checking, rule-merging validation, and structural invariant checking. We evaluate BorrowMIR by encoding Stacked Borrows, Tree Borrows, and smaller model variants. The generated full models match native Miri outcomes on the Miri borrow-tracker tests and selected Rust standard-library tests. Mutation testing detects 67 of 70 injected rule faults, and the BorrowMIR specifications are substantially smaller than the corresponding native implementations. Runtime overhead is the main limitation: generated Stacked Borrows is usable on the selected test suite, while generated Tree Borrows is considerably slower. Our results indicate that BorrowMIR supports the definition, execution, comparison, and bounded validation of Rust aliasing models outside Miri's native borrow-tracker implementation.
...
Rust's ownership-based type system provides strong memory-safety and aliasing guarantees, but unsafe Rust allows raw pointers to alias with references, which the static borrow checker cannot track. Dynamic aliasing models such as Stacked Borrows and Tree Borrows address this gap in the Rust interpreter Miri by defining when pointer uses create undefined behaviour. However, these models are implemented directly inside Miri, so changing or comparing model variants requires modifying the interpreter and risks semantic drift between model definitions and implementations. We present BorrowMIR, a model-parametric framework for specifying Rust aliasing models as explicit state-transition rules over MIR-level borrow events. BorrowMIR represents borrow state using per-location directed acyclic graphs with allocation-level metadata, allowing stack-shaped, tree-shaped, and more general alias derivation structures. From a single model specification, BorrowMIR generates a Rust backend for execution inside Miri and a TLA+ backend for trace checking, rule-merging validation, and structural invariant checking. We evaluate BorrowMIR by encoding Stacked Borrows, Tree Borrows, and smaller model variants. The generated full models match native Miri outcomes on the Miri borrow-tracker tests and selected Rust standard-library tests. Mutation testing detects 67 of 70 injected rule faults, and the BorrowMIR specifications are substantially smaller than the corresponding native implementations. Runtime overhead is the main limitation: generated Stacked Borrows is usable on the selected test suite, while generated Tree Borrows is considerably slower. Our results indicate that BorrowMIR supports the definition, execution, comparison, and bounded validation of Rust aliasing models outside Miri's native borrow-tracker implementation.
Exploiting landing in multi-UAV systems for enhanced cooperative localization
A Gaussian Belief Propagation Perspective
Master thesis
(2026)
-
S.H. Molenkamp, K.G. Langendoen, R.T. Rajan, Pieter Simke de Vries, G.B.G. Potter
Multi-agent Systems increasingly rely on cooperative localisation to navigate GNSS-denied environments using only relative measurements. However, these purely relative networks inevitably accumulate global trajectory drift over time. This thesis investigates whether strategically landing a UAV to serve as a stationary anchor can mitigate this accumulated drift and improve the absolute localization performance of the overarching MAS.
Unlike traditional filtering methods that only estimate the current state, this work adopts Gaussian Belief Propagation to achieve scalable, fully distributed estimation, solving factor graphs locally to jointly optimize the trajectory history. Within this distributed architecture, low-cost IMU and range-bearing constraints were simulated to evaluate various landed anchor mathematical representations (Unary, Persistent Variable, and ZUPT) across continuous and multi-group flight topologies. The results demonstrate that in single-group, continuous flight scenarios, landing a drone fails to bound global drift because the dense network topology becomes highly overconfident and rigidly locks the anchor into a drifted state. Conversely, when applied to multi-group deployments, representing the landed anchor with a Zero-Velocity Update (ZUPT) model mathematically combats this overconfidence through artificial covariance inflation, allowing the anchor to absorb geometric corrections and successfully reset the drift of subsequent passing groups.
The findings imply that autonomous MAS should not simply deploy stationary anchors during continuous, dense flight without risking algorithmic divergence; instead, operators must explicitly structure missions into temporal, multi-wave batches to safely exploit this relative infrastructure. While the baseline GBP architecture was verified using real-world datasets, the conclusions regarding landed anchors rely on synthetic evaluations within a 2D simulation, meaning the proposed system must still be validated against the physical hardware and 3D flight complexities of real-world deployments. ...
Unlike traditional filtering methods that only estimate the current state, this work adopts Gaussian Belief Propagation to achieve scalable, fully distributed estimation, solving factor graphs locally to jointly optimize the trajectory history. Within this distributed architecture, low-cost IMU and range-bearing constraints were simulated to evaluate various landed anchor mathematical representations (Unary, Persistent Variable, and ZUPT) across continuous and multi-group flight topologies. The results demonstrate that in single-group, continuous flight scenarios, landing a drone fails to bound global drift because the dense network topology becomes highly overconfident and rigidly locks the anchor into a drifted state. Conversely, when applied to multi-group deployments, representing the landed anchor with a Zero-Velocity Update (ZUPT) model mathematically combats this overconfidence through artificial covariance inflation, allowing the anchor to absorb geometric corrections and successfully reset the drift of subsequent passing groups.
The findings imply that autonomous MAS should not simply deploy stationary anchors during continuous, dense flight without risking algorithmic divergence; instead, operators must explicitly structure missions into temporal, multi-wave batches to safely exploit this relative infrastructure. While the baseline GBP architecture was verified using real-world datasets, the conclusions regarding landed anchors rely on synthetic evaluations within a 2D simulation, meaning the proposed system must still be validated against the physical hardware and 3D flight complexities of real-world deployments. ...
Multi-agent Systems increasingly rely on cooperative localisation to navigate GNSS-denied environments using only relative measurements. However, these purely relative networks inevitably accumulate global trajectory drift over time. This thesis investigates whether strategically landing a UAV to serve as a stationary anchor can mitigate this accumulated drift and improve the absolute localization performance of the overarching MAS.
Unlike traditional filtering methods that only estimate the current state, this work adopts Gaussian Belief Propagation to achieve scalable, fully distributed estimation, solving factor graphs locally to jointly optimize the trajectory history. Within this distributed architecture, low-cost IMU and range-bearing constraints were simulated to evaluate various landed anchor mathematical representations (Unary, Persistent Variable, and ZUPT) across continuous and multi-group flight topologies. The results demonstrate that in single-group, continuous flight scenarios, landing a drone fails to bound global drift because the dense network topology becomes highly overconfident and rigidly locks the anchor into a drifted state. Conversely, when applied to multi-group deployments, representing the landed anchor with a Zero-Velocity Update (ZUPT) model mathematically combats this overconfidence through artificial covariance inflation, allowing the anchor to absorb geometric corrections and successfully reset the drift of subsequent passing groups.
The findings imply that autonomous MAS should not simply deploy stationary anchors during continuous, dense flight without risking algorithmic divergence; instead, operators must explicitly structure missions into temporal, multi-wave batches to safely exploit this relative infrastructure. While the baseline GBP architecture was verified using real-world datasets, the conclusions regarding landed anchors rely on synthetic evaluations within a 2D simulation, meaning the proposed system must still be validated against the physical hardware and 3D flight complexities of real-world deployments.
Unlike traditional filtering methods that only estimate the current state, this work adopts Gaussian Belief Propagation to achieve scalable, fully distributed estimation, solving factor graphs locally to jointly optimize the trajectory history. Within this distributed architecture, low-cost IMU and range-bearing constraints were simulated to evaluate various landed anchor mathematical representations (Unary, Persistent Variable, and ZUPT) across continuous and multi-group flight topologies. The results demonstrate that in single-group, continuous flight scenarios, landing a drone fails to bound global drift because the dense network topology becomes highly overconfident and rigidly locks the anchor into a drifted state. Conversely, when applied to multi-group deployments, representing the landed anchor with a Zero-Velocity Update (ZUPT) model mathematically combats this overconfidence through artificial covariance inflation, allowing the anchor to absorb geometric corrections and successfully reset the drift of subsequent passing groups.
The findings imply that autonomous MAS should not simply deploy stationary anchors during continuous, dense flight without risking algorithmic divergence; instead, operators must explicitly structure missions into temporal, multi-wave batches to safely exploit this relative infrastructure. While the baseline GBP architecture was verified using real-world datasets, the conclusions regarding landed anchors rely on synthetic evaluations within a 2D simulation, meaning the proposed system must still be validated against the physical hardware and 3D flight complexities of real-world deployments.
Eye gaze contains rich information about human attention and cognitive processes. This capability makes the underlying technology, known as gaze estimation, a critical enabler for many applications, ranging from human-computer interaction to cognitive sensing systems. With the development of deep learning, appearance-based gaze estimation has emerged as a promising solution due to its capability of using general-purpose cameras for non-intrusive and cost-effective gaze estimation.
To build applications based on appearance-based gaze estimation, developers can choose among three paradigms. One paradigm is to train gaze estimation models themselves, which allows developers to customize models to meet various application requirements. Another option is to adopt pre-trained gaze estimation models, which avoids the resource-intensive process for model training. The third paradigm is to call gaze estimation services running on the cloud, which are well-suited for developers who wish to reduce the resource consumption for model deployment. In this case, the full-face images of users are sent to the service provider, which returns estimated gaze directions.
Despite these paradigms offering flexible options to developers for building applications, each paradigm comes with distinct challenges that hinder widespread adoption. Training an accurate gaze estimation model requires the availability of large-scale gaze datasets and the adoption of complex neural networks. The former is sparse and difficult to collect, while the latter demands substantial computational resources. Adopting pre-trained models removes the resource burden of model training, but exposes gaze estimation systems to backdoor attacks, in which an adversary can inject a backdoor into the pre-trained model and manipulate its output with a visual trigger after deployment. This compromises the security of many gaze-based applications, e.g., causing the driving assistant system to fail in tracking the driver’s attention. Lastly, calling gaze estimation services raises severe privacy concerns. This is because these services often operate as black boxes, leaving users unaware of how their face images that contain sensitive attributes are processed or utilized.
Taking these paradigms together, we observe that they either require substantial resources for model training or raise trustworthiness concerns due to the involvement of third parties. This motivates the main research question of this dissertation: “How can we make gaze estimation systems both resource-efficient and trustworthy? ” This dissertation answers this question by addressing the challenges associated with each paradigm.
To reduce the resource burden of self-trained models, we present a resource-efficient framework that includes frequency-domain gaze estimation and gaze-aware contrastive learning. The frequency-domain gaze estimation exploits the feature extraction capability and the spectral compaction property of the discrete cosine transform to substantially reduce the computational cost of gaze estimation models. Meanwhile, gaze-aware contrastive learning enables learning gaze representations in an unsupervised manner to overcome the data labeling hurdle. We show that the proposed framework can achieve comparable gaze estimation performance to existing approaches that rely on a largescale, well-labeled dataset, while enabling up to 1.67 times speedup in inference latency.
For pre-trained gaze estimation models, we explore solutions to defend against backdoor attacks. We identify the key characteristics that distinguish backdoored gaze estimation models from benign ones, based on which we propose a novel approach to reverse-engineer the backdoor trigger that leads to the identified characteristics. Given a pre-trained model, we use the reverse-engineered trigger to determine whether it is backdoored or not. If it is identified as a compromised model, we further use the reverse engineered trigger to mitigate its backdoor behavior. We show that the proposed method can defend against various backdoor attacks.
To address privacy concerns in gaze estimation services, we develop a privacy preserver that converts privacy-sensitive full-face images into obfuscated images. The obfuscated versions are then shared with the service provider for gaze estimation. The privacy preserver is designed to generate obfuscated images that exhibit the same facial appearance for different users to protect user privacy, while preserving the gaze features of the raw images to remain effective for accurate gaze estimation. Our experiments show that obfuscated images can effectively protect user privacy while leading to comparable gaze estimation performance to the original images.
Overall, this dissertation contributes to the development of resource-efficient and trustworthy gaze estimation systems. We enhance the resource efficiency of using self-trained models, which typically demand substantial resources, while improving trustworthiness of the other two paradigms, where the resource burden is offloaded to external parties through the use of pre-trained models or vendor-provided services.
...
To build applications based on appearance-based gaze estimation, developers can choose among three paradigms. One paradigm is to train gaze estimation models themselves, which allows developers to customize models to meet various application requirements. Another option is to adopt pre-trained gaze estimation models, which avoids the resource-intensive process for model training. The third paradigm is to call gaze estimation services running on the cloud, which are well-suited for developers who wish to reduce the resource consumption for model deployment. In this case, the full-face images of users are sent to the service provider, which returns estimated gaze directions.
Despite these paradigms offering flexible options to developers for building applications, each paradigm comes with distinct challenges that hinder widespread adoption. Training an accurate gaze estimation model requires the availability of large-scale gaze datasets and the adoption of complex neural networks. The former is sparse and difficult to collect, while the latter demands substantial computational resources. Adopting pre-trained models removes the resource burden of model training, but exposes gaze estimation systems to backdoor attacks, in which an adversary can inject a backdoor into the pre-trained model and manipulate its output with a visual trigger after deployment. This compromises the security of many gaze-based applications, e.g., causing the driving assistant system to fail in tracking the driver’s attention. Lastly, calling gaze estimation services raises severe privacy concerns. This is because these services often operate as black boxes, leaving users unaware of how their face images that contain sensitive attributes are processed or utilized.
Taking these paradigms together, we observe that they either require substantial resources for model training or raise trustworthiness concerns due to the involvement of third parties. This motivates the main research question of this dissertation: “How can we make gaze estimation systems both resource-efficient and trustworthy? ” This dissertation answers this question by addressing the challenges associated with each paradigm.
To reduce the resource burden of self-trained models, we present a resource-efficient framework that includes frequency-domain gaze estimation and gaze-aware contrastive learning. The frequency-domain gaze estimation exploits the feature extraction capability and the spectral compaction property of the discrete cosine transform to substantially reduce the computational cost of gaze estimation models. Meanwhile, gaze-aware contrastive learning enables learning gaze representations in an unsupervised manner to overcome the data labeling hurdle. We show that the proposed framework can achieve comparable gaze estimation performance to existing approaches that rely on a largescale, well-labeled dataset, while enabling up to 1.67 times speedup in inference latency.
For pre-trained gaze estimation models, we explore solutions to defend against backdoor attacks. We identify the key characteristics that distinguish backdoored gaze estimation models from benign ones, based on which we propose a novel approach to reverse-engineer the backdoor trigger that leads to the identified characteristics. Given a pre-trained model, we use the reverse-engineered trigger to determine whether it is backdoored or not. If it is identified as a compromised model, we further use the reverse engineered trigger to mitigate its backdoor behavior. We show that the proposed method can defend against various backdoor attacks.
To address privacy concerns in gaze estimation services, we develop a privacy preserver that converts privacy-sensitive full-face images into obfuscated images. The obfuscated versions are then shared with the service provider for gaze estimation. The privacy preserver is designed to generate obfuscated images that exhibit the same facial appearance for different users to protect user privacy, while preserving the gaze features of the raw images to remain effective for accurate gaze estimation. Our experiments show that obfuscated images can effectively protect user privacy while leading to comparable gaze estimation performance to the original images.
Overall, this dissertation contributes to the development of resource-efficient and trustworthy gaze estimation systems. We enhance the resource efficiency of using self-trained models, which typically demand substantial resources, while improving trustworthiness of the other two paradigms, where the resource burden is offloaded to external parties through the use of pre-trained models or vendor-provided services.
...
Eye gaze contains rich information about human attention and cognitive processes. This capability makes the underlying technology, known as gaze estimation, a critical enabler for many applications, ranging from human-computer interaction to cognitive sensing systems. With the development of deep learning, appearance-based gaze estimation has emerged as a promising solution due to its capability of using general-purpose cameras for non-intrusive and cost-effective gaze estimation.
To build applications based on appearance-based gaze estimation, developers can choose among three paradigms. One paradigm is to train gaze estimation models themselves, which allows developers to customize models to meet various application requirements. Another option is to adopt pre-trained gaze estimation models, which avoids the resource-intensive process for model training. The third paradigm is to call gaze estimation services running on the cloud, which are well-suited for developers who wish to reduce the resource consumption for model deployment. In this case, the full-face images of users are sent to the service provider, which returns estimated gaze directions.
Despite these paradigms offering flexible options to developers for building applications, each paradigm comes with distinct challenges that hinder widespread adoption. Training an accurate gaze estimation model requires the availability of large-scale gaze datasets and the adoption of complex neural networks. The former is sparse and difficult to collect, while the latter demands substantial computational resources. Adopting pre-trained models removes the resource burden of model training, but exposes gaze estimation systems to backdoor attacks, in which an adversary can inject a backdoor into the pre-trained model and manipulate its output with a visual trigger after deployment. This compromises the security of many gaze-based applications, e.g., causing the driving assistant system to fail in tracking the driver’s attention. Lastly, calling gaze estimation services raises severe privacy concerns. This is because these services often operate as black boxes, leaving users unaware of how their face images that contain sensitive attributes are processed or utilized.
Taking these paradigms together, we observe that they either require substantial resources for model training or raise trustworthiness concerns due to the involvement of third parties. This motivates the main research question of this dissertation: “How can we make gaze estimation systems both resource-efficient and trustworthy? ” This dissertation answers this question by addressing the challenges associated with each paradigm.
To reduce the resource burden of self-trained models, we present a resource-efficient framework that includes frequency-domain gaze estimation and gaze-aware contrastive learning. The frequency-domain gaze estimation exploits the feature extraction capability and the spectral compaction property of the discrete cosine transform to substantially reduce the computational cost of gaze estimation models. Meanwhile, gaze-aware contrastive learning enables learning gaze representations in an unsupervised manner to overcome the data labeling hurdle. We show that the proposed framework can achieve comparable gaze estimation performance to existing approaches that rely on a largescale, well-labeled dataset, while enabling up to 1.67 times speedup in inference latency.
For pre-trained gaze estimation models, we explore solutions to defend against backdoor attacks. We identify the key characteristics that distinguish backdoored gaze estimation models from benign ones, based on which we propose a novel approach to reverse-engineer the backdoor trigger that leads to the identified characteristics. Given a pre-trained model, we use the reverse-engineered trigger to determine whether it is backdoored or not. If it is identified as a compromised model, we further use the reverse engineered trigger to mitigate its backdoor behavior. We show that the proposed method can defend against various backdoor attacks.
To address privacy concerns in gaze estimation services, we develop a privacy preserver that converts privacy-sensitive full-face images into obfuscated images. The obfuscated versions are then shared with the service provider for gaze estimation. The privacy preserver is designed to generate obfuscated images that exhibit the same facial appearance for different users to protect user privacy, while preserving the gaze features of the raw images to remain effective for accurate gaze estimation. Our experiments show that obfuscated images can effectively protect user privacy while leading to comparable gaze estimation performance to the original images.
Overall, this dissertation contributes to the development of resource-efficient and trustworthy gaze estimation systems. We enhance the resource efficiency of using self-trained models, which typically demand substantial resources, while improving trustworthiness of the other two paradigms, where the resource burden is offloaded to external parties through the use of pre-trained models or vendor-provided services.
To build applications based on appearance-based gaze estimation, developers can choose among three paradigms. One paradigm is to train gaze estimation models themselves, which allows developers to customize models to meet various application requirements. Another option is to adopt pre-trained gaze estimation models, which avoids the resource-intensive process for model training. The third paradigm is to call gaze estimation services running on the cloud, which are well-suited for developers who wish to reduce the resource consumption for model deployment. In this case, the full-face images of users are sent to the service provider, which returns estimated gaze directions.
Despite these paradigms offering flexible options to developers for building applications, each paradigm comes with distinct challenges that hinder widespread adoption. Training an accurate gaze estimation model requires the availability of large-scale gaze datasets and the adoption of complex neural networks. The former is sparse and difficult to collect, while the latter demands substantial computational resources. Adopting pre-trained models removes the resource burden of model training, but exposes gaze estimation systems to backdoor attacks, in which an adversary can inject a backdoor into the pre-trained model and manipulate its output with a visual trigger after deployment. This compromises the security of many gaze-based applications, e.g., causing the driving assistant system to fail in tracking the driver’s attention. Lastly, calling gaze estimation services raises severe privacy concerns. This is because these services often operate as black boxes, leaving users unaware of how their face images that contain sensitive attributes are processed or utilized.
Taking these paradigms together, we observe that they either require substantial resources for model training or raise trustworthiness concerns due to the involvement of third parties. This motivates the main research question of this dissertation: “How can we make gaze estimation systems both resource-efficient and trustworthy? ” This dissertation answers this question by addressing the challenges associated with each paradigm.
To reduce the resource burden of self-trained models, we present a resource-efficient framework that includes frequency-domain gaze estimation and gaze-aware contrastive learning. The frequency-domain gaze estimation exploits the feature extraction capability and the spectral compaction property of the discrete cosine transform to substantially reduce the computational cost of gaze estimation models. Meanwhile, gaze-aware contrastive learning enables learning gaze representations in an unsupervised manner to overcome the data labeling hurdle. We show that the proposed framework can achieve comparable gaze estimation performance to existing approaches that rely on a largescale, well-labeled dataset, while enabling up to 1.67 times speedup in inference latency.
For pre-trained gaze estimation models, we explore solutions to defend against backdoor attacks. We identify the key characteristics that distinguish backdoored gaze estimation models from benign ones, based on which we propose a novel approach to reverse-engineer the backdoor trigger that leads to the identified characteristics. Given a pre-trained model, we use the reverse-engineered trigger to determine whether it is backdoored or not. If it is identified as a compromised model, we further use the reverse engineered trigger to mitigate its backdoor behavior. We show that the proposed method can defend against various backdoor attacks.
To address privacy concerns in gaze estimation services, we develop a privacy preserver that converts privacy-sensitive full-face images into obfuscated images. The obfuscated versions are then shared with the service provider for gaze estimation. The privacy preserver is designed to generate obfuscated images that exhibit the same facial appearance for different users to protect user privacy, while preserving the gaze features of the raw images to remain effective for accurate gaze estimation. Our experiments show that obfuscated images can effectively protect user privacy while leading to comparable gaze estimation performance to the original images.
Overall, this dissertation contributes to the development of resource-efficient and trustworthy gaze estimation systems. We enhance the resource efficiency of using self-trained models, which typically demand substantial resources, while improving trustworthiness of the other two paradigms, where the resource burden is offloaded to external parties through the use of pre-trained models or vendor-provided services.
After the potential of this work is realized, people will be able to physically manipulate remote environments. For example, a skilled artist in Tokyo could paint delicate calligraphy on a canvas in Paris, feeling each stroke as if they were local. A surgeon in London could operate on a patient in a remote village, sensing the precise resistance of tissue through robotic instruments. A firefighter in Los Angeles could save people from a burning building without the need to put his own life at stake. Extending our human touch across great distances opens doors to new forms of work, collaboration, and human connection without needing physical presence.
Realizing this vision requires the successful implementation of Haptic Bilateral Teleoperation (HBT). An HBT system must fulfill two core requirements: precise replication of the operator’s actions by a remote robot and accurate, responsive feedback to guide those actions. These requirements are inherently subjective, varying across individuals, tasks, and applications, adding significant complexity to both the system design and evaluation. At first glance, realizing HBT may seem an insurmountable challenge. Conventional wisdom suggests that the stringent network requirements, such as ultra-low latency and near-perfect reliability, far exceed the capabilities of current network technology. The latency constraints are so strict that even fundamental physical limits, such as the speed of light, impose onerous restrictions on the maximum feasible distance between the operator and the remote environment.
Overcoming these challenges demands a holistic approach. On the one hand, we must push network technology to its limits, striving for lower latency, higher reliability, and optimized communication protocols explicitly tailored for HBT applications. On the other hand, we must also explore alternative approaches that lower the network requirements of HBT systems, especially the latency requirement. For both of these directions, it is essential to have a deep understanding of the entire HBT system, particularly the role of the human operator. Unlike most systems, where performance is measured through objective metrics, HBT introduces a distinctive challenge: HBT systems must be designed for both technical performance and the user’s subjective experience.
In this dissertation, we first provide a deeper understanding of HBT systems and examine how network behavior influences user experience. In particular, we identify the underlying reasons behind the stringent network requirements. First, through multiple repeated user studies, we demonstrate that the reliability of the kinematic demands and force modalities is low, especially at the packet rate 1 kHz. Even with 50%, packet loss, we demonstrate that users are largely unaffected due to strong temporal correlation in these modalities.
More importantly, we pinpoint the fundamental cause of the strict low-latency requirement. It is not merely the presence of delay but rather the unintended forces that arise due to the combination of active force feedback and a closed-loop control system. This interaction is unique because users do not perceive latency directly. Instead, they experience the resulting unnatural forces.
Because the main cause for the stringent network requirements is so specific, it provides a clear target for research. Next, we explore multiple approaches to address this particular interaction, which is the primary source of stringent latency constraints. First, we optimize the MAC protocols with a strict focus on minimizing latency for both the kinematic and force modalities. Next, we investigate methods to manipulate the transmitted data in a way that does not impede the human operator, aiming to mitigate the adverse effects of network latency on force feedback. Finally, we take a more radical approach by moving away from direct transmission of force feedback altogether, instead leveraging predictive models to estimate force feedback locally.
An important insight from this dissertation is the path forward for HBT systems. Future HBT systems should integrate predictive force feedback with live video transmission, leveraging the advantages of each modality. Predictive force feedback offers a viable alternative to the stringent latency constraints of transmitted force feedback. Minor inaccuracies in force feedback are often imperceptible to human operators. Meanwhile, live video transmission circumvents the complexities of visual prediction while operating within a latency range of approximately 100ms. This is significantly more feasible than the 1ms latency required for direct force feedback transmissions.
This dissertation has three important takeaways. First, it provides a deeper understanding of how network performance shapes user experience in HBT. Second, it demonstrates alternative approaches that enable HBT beyond direct network improvements. Third, it proposes a path forward that integrates live video with predictive force feedback. Despite these advancements, significant challenges remain. Scaling HBT to highly dynamic environments, where unpredictability complicates prediction of force feedback, remains a major hurdle. Additionally, managing discrepancies between the operator’s predictive experience and the actual remote events is crucial to maintaining intuitive and stable interactions. While these challenges persist, none appear insurmountable. With continued progress, HBT can become a transformative technology, opening doors to new forms of work, collaboration, and human connection without needing physical presence.
...
Realizing this vision requires the successful implementation of Haptic Bilateral Teleoperation (HBT). An HBT system must fulfill two core requirements: precise replication of the operator’s actions by a remote robot and accurate, responsive feedback to guide those actions. These requirements are inherently subjective, varying across individuals, tasks, and applications, adding significant complexity to both the system design and evaluation. At first glance, realizing HBT may seem an insurmountable challenge. Conventional wisdom suggests that the stringent network requirements, such as ultra-low latency and near-perfect reliability, far exceed the capabilities of current network technology. The latency constraints are so strict that even fundamental physical limits, such as the speed of light, impose onerous restrictions on the maximum feasible distance between the operator and the remote environment.
Overcoming these challenges demands a holistic approach. On the one hand, we must push network technology to its limits, striving for lower latency, higher reliability, and optimized communication protocols explicitly tailored for HBT applications. On the other hand, we must also explore alternative approaches that lower the network requirements of HBT systems, especially the latency requirement. For both of these directions, it is essential to have a deep understanding of the entire HBT system, particularly the role of the human operator. Unlike most systems, where performance is measured through objective metrics, HBT introduces a distinctive challenge: HBT systems must be designed for both technical performance and the user’s subjective experience.
In this dissertation, we first provide a deeper understanding of HBT systems and examine how network behavior influences user experience. In particular, we identify the underlying reasons behind the stringent network requirements. First, through multiple repeated user studies, we demonstrate that the reliability of the kinematic demands and force modalities is low, especially at the packet rate 1 kHz. Even with 50%, packet loss, we demonstrate that users are largely unaffected due to strong temporal correlation in these modalities.
More importantly, we pinpoint the fundamental cause of the strict low-latency requirement. It is not merely the presence of delay but rather the unintended forces that arise due to the combination of active force feedback and a closed-loop control system. This interaction is unique because users do not perceive latency directly. Instead, they experience the resulting unnatural forces.
Because the main cause for the stringent network requirements is so specific, it provides a clear target for research. Next, we explore multiple approaches to address this particular interaction, which is the primary source of stringent latency constraints. First, we optimize the MAC protocols with a strict focus on minimizing latency for both the kinematic and force modalities. Next, we investigate methods to manipulate the transmitted data in a way that does not impede the human operator, aiming to mitigate the adverse effects of network latency on force feedback. Finally, we take a more radical approach by moving away from direct transmission of force feedback altogether, instead leveraging predictive models to estimate force feedback locally.
An important insight from this dissertation is the path forward for HBT systems. Future HBT systems should integrate predictive force feedback with live video transmission, leveraging the advantages of each modality. Predictive force feedback offers a viable alternative to the stringent latency constraints of transmitted force feedback. Minor inaccuracies in force feedback are often imperceptible to human operators. Meanwhile, live video transmission circumvents the complexities of visual prediction while operating within a latency range of approximately 100ms. This is significantly more feasible than the 1ms latency required for direct force feedback transmissions.
This dissertation has three important takeaways. First, it provides a deeper understanding of how network performance shapes user experience in HBT. Second, it demonstrates alternative approaches that enable HBT beyond direct network improvements. Third, it proposes a path forward that integrates live video with predictive force feedback. Despite these advancements, significant challenges remain. Scaling HBT to highly dynamic environments, where unpredictability complicates prediction of force feedback, remains a major hurdle. Additionally, managing discrepancies between the operator’s predictive experience and the actual remote events is crucial to maintaining intuitive and stable interactions. While these challenges persist, none appear insurmountable. With continued progress, HBT can become a transformative technology, opening doors to new forms of work, collaboration, and human connection without needing physical presence.
...
After the potential of this work is realized, people will be able to physically manipulate remote environments. For example, a skilled artist in Tokyo could paint delicate calligraphy on a canvas in Paris, feeling each stroke as if they were local. A surgeon in London could operate on a patient in a remote village, sensing the precise resistance of tissue through robotic instruments. A firefighter in Los Angeles could save people from a burning building without the need to put his own life at stake. Extending our human touch across great distances opens doors to new forms of work, collaboration, and human connection without needing physical presence.
Realizing this vision requires the successful implementation of Haptic Bilateral Teleoperation (HBT). An HBT system must fulfill two core requirements: precise replication of the operator’s actions by a remote robot and accurate, responsive feedback to guide those actions. These requirements are inherently subjective, varying across individuals, tasks, and applications, adding significant complexity to both the system design and evaluation. At first glance, realizing HBT may seem an insurmountable challenge. Conventional wisdom suggests that the stringent network requirements, such as ultra-low latency and near-perfect reliability, far exceed the capabilities of current network technology. The latency constraints are so strict that even fundamental physical limits, such as the speed of light, impose onerous restrictions on the maximum feasible distance between the operator and the remote environment.
Overcoming these challenges demands a holistic approach. On the one hand, we must push network technology to its limits, striving for lower latency, higher reliability, and optimized communication protocols explicitly tailored for HBT applications. On the other hand, we must also explore alternative approaches that lower the network requirements of HBT systems, especially the latency requirement. For both of these directions, it is essential to have a deep understanding of the entire HBT system, particularly the role of the human operator. Unlike most systems, where performance is measured through objective metrics, HBT introduces a distinctive challenge: HBT systems must be designed for both technical performance and the user’s subjective experience.
In this dissertation, we first provide a deeper understanding of HBT systems and examine how network behavior influences user experience. In particular, we identify the underlying reasons behind the stringent network requirements. First, through multiple repeated user studies, we demonstrate that the reliability of the kinematic demands and force modalities is low, especially at the packet rate 1 kHz. Even with 50%, packet loss, we demonstrate that users are largely unaffected due to strong temporal correlation in these modalities.
More importantly, we pinpoint the fundamental cause of the strict low-latency requirement. It is not merely the presence of delay but rather the unintended forces that arise due to the combination of active force feedback and a closed-loop control system. This interaction is unique because users do not perceive latency directly. Instead, they experience the resulting unnatural forces.
Because the main cause for the stringent network requirements is so specific, it provides a clear target for research. Next, we explore multiple approaches to address this particular interaction, which is the primary source of stringent latency constraints. First, we optimize the MAC protocols with a strict focus on minimizing latency for both the kinematic and force modalities. Next, we investigate methods to manipulate the transmitted data in a way that does not impede the human operator, aiming to mitigate the adverse effects of network latency on force feedback. Finally, we take a more radical approach by moving away from direct transmission of force feedback altogether, instead leveraging predictive models to estimate force feedback locally.
An important insight from this dissertation is the path forward for HBT systems. Future HBT systems should integrate predictive force feedback with live video transmission, leveraging the advantages of each modality. Predictive force feedback offers a viable alternative to the stringent latency constraints of transmitted force feedback. Minor inaccuracies in force feedback are often imperceptible to human operators. Meanwhile, live video transmission circumvents the complexities of visual prediction while operating within a latency range of approximately 100ms. This is significantly more feasible than the 1ms latency required for direct force feedback transmissions.
This dissertation has three important takeaways. First, it provides a deeper understanding of how network performance shapes user experience in HBT. Second, it demonstrates alternative approaches that enable HBT beyond direct network improvements. Third, it proposes a path forward that integrates live video with predictive force feedback. Despite these advancements, significant challenges remain. Scaling HBT to highly dynamic environments, where unpredictability complicates prediction of force feedback, remains a major hurdle. Additionally, managing discrepancies between the operator’s predictive experience and the actual remote events is crucial to maintaining intuitive and stable interactions. While these challenges persist, none appear insurmountable. With continued progress, HBT can become a transformative technology, opening doors to new forms of work, collaboration, and human connection without needing physical presence.
Realizing this vision requires the successful implementation of Haptic Bilateral Teleoperation (HBT). An HBT system must fulfill two core requirements: precise replication of the operator’s actions by a remote robot and accurate, responsive feedback to guide those actions. These requirements are inherently subjective, varying across individuals, tasks, and applications, adding significant complexity to both the system design and evaluation. At first glance, realizing HBT may seem an insurmountable challenge. Conventional wisdom suggests that the stringent network requirements, such as ultra-low latency and near-perfect reliability, far exceed the capabilities of current network technology. The latency constraints are so strict that even fundamental physical limits, such as the speed of light, impose onerous restrictions on the maximum feasible distance between the operator and the remote environment.
Overcoming these challenges demands a holistic approach. On the one hand, we must push network technology to its limits, striving for lower latency, higher reliability, and optimized communication protocols explicitly tailored for HBT applications. On the other hand, we must also explore alternative approaches that lower the network requirements of HBT systems, especially the latency requirement. For both of these directions, it is essential to have a deep understanding of the entire HBT system, particularly the role of the human operator. Unlike most systems, where performance is measured through objective metrics, HBT introduces a distinctive challenge: HBT systems must be designed for both technical performance and the user’s subjective experience.
In this dissertation, we first provide a deeper understanding of HBT systems and examine how network behavior influences user experience. In particular, we identify the underlying reasons behind the stringent network requirements. First, through multiple repeated user studies, we demonstrate that the reliability of the kinematic demands and force modalities is low, especially at the packet rate 1 kHz. Even with 50%, packet loss, we demonstrate that users are largely unaffected due to strong temporal correlation in these modalities.
More importantly, we pinpoint the fundamental cause of the strict low-latency requirement. It is not merely the presence of delay but rather the unintended forces that arise due to the combination of active force feedback and a closed-loop control system. This interaction is unique because users do not perceive latency directly. Instead, they experience the resulting unnatural forces.
Because the main cause for the stringent network requirements is so specific, it provides a clear target for research. Next, we explore multiple approaches to address this particular interaction, which is the primary source of stringent latency constraints. First, we optimize the MAC protocols with a strict focus on minimizing latency for both the kinematic and force modalities. Next, we investigate methods to manipulate the transmitted data in a way that does not impede the human operator, aiming to mitigate the adverse effects of network latency on force feedback. Finally, we take a more radical approach by moving away from direct transmission of force feedback altogether, instead leveraging predictive models to estimate force feedback locally.
An important insight from this dissertation is the path forward for HBT systems. Future HBT systems should integrate predictive force feedback with live video transmission, leveraging the advantages of each modality. Predictive force feedback offers a viable alternative to the stringent latency constraints of transmitted force feedback. Minor inaccuracies in force feedback are often imperceptible to human operators. Meanwhile, live video transmission circumvents the complexities of visual prediction while operating within a latency range of approximately 100ms. This is significantly more feasible than the 1ms latency required for direct force feedback transmissions.
This dissertation has three important takeaways. First, it provides a deeper understanding of how network performance shapes user experience in HBT. Second, it demonstrates alternative approaches that enable HBT beyond direct network improvements. Third, it proposes a path forward that integrates live video with predictive force feedback. Despite these advancements, significant challenges remain. Scaling HBT to highly dynamic environments, where unpredictability complicates prediction of force feedback, remains a major hurdle. Additionally, managing discrepancies between the operator’s predictive experience and the actual remote events is crucial to maintaining intuitive and stable interactions. While these challenges persist, none appear insurmountable. With continued progress, HBT can become a transformative technology, opening doors to new forms of work, collaboration, and human connection without needing physical presence.
This dissertation is about translating concurrent programs between computer architectures. Legacy programs—built-for and tested-on x86—behave differently on newer architectures, such as Arm and RISC-V. Particularly, weak memory behaviors emerge when two micro-architectural features interact: (i) concurrency, where multiple CPU cores simultaneously execute parts of a program, and (ii) out-of-order execution, where a CPU core reorders instructions to increase throughput. Programs can non-deterministically show one of various weak memory behaviors, meaning it could behave differently when executing again. Those behaviors differ between architectures. When migrating programs from x86 to Arm or RISC-V, the same program could non-deterministically show behaviors never observed on x86.
In part I, we look at binary translators, which are software systems that translate compiled binary programs between architectures. We study the translation process of three such real-world systems, identify errors in their translation of concurrency primitives, and fix them. We propose mathematically-rigorous weak memory models for these translators. We then define mapping schemes to translate concurrency primitives one-by-one from x86 to Arm and RISC-V. With the formal semantics, we prove those mapping schemes correct in the Agda proof assistant.
In part II, we study the common structure of our weak memory proofs written in Agda. As those proofs are often large, complex, and rigid, we identify their common structures for which we identify domain-specific abstractions. We implement those abstractions in our novel Agda proof framework Burrow to greatly simplify writing future weak memory proofs.
In part III, we use dynamic analysis to identify weak behaviors that were never seen on x86 but could appear on Arm. Our analysis simulates the program’s execution with the formal weak memory semantics of x86 and Arm. This analysis identifies only the new behaviors the program shows in practice. After finding any new behavior on Arm, we judiciously modify the program to eliminate only that behavior. ...
In part I, we look at binary translators, which are software systems that translate compiled binary programs between architectures. We study the translation process of three such real-world systems, identify errors in their translation of concurrency primitives, and fix them. We propose mathematically-rigorous weak memory models for these translators. We then define mapping schemes to translate concurrency primitives one-by-one from x86 to Arm and RISC-V. With the formal semantics, we prove those mapping schemes correct in the Agda proof assistant.
In part II, we study the common structure of our weak memory proofs written in Agda. As those proofs are often large, complex, and rigid, we identify their common structures for which we identify domain-specific abstractions. We implement those abstractions in our novel Agda proof framework Burrow to greatly simplify writing future weak memory proofs.
In part III, we use dynamic analysis to identify weak behaviors that were never seen on x86 but could appear on Arm. Our analysis simulates the program’s execution with the formal weak memory semantics of x86 and Arm. This analysis identifies only the new behaviors the program shows in practice. After finding any new behavior on Arm, we judiciously modify the program to eliminate only that behavior. ...
This dissertation is about translating concurrent programs between computer architectures. Legacy programs—built-for and tested-on x86—behave differently on newer architectures, such as Arm and RISC-V. Particularly, weak memory behaviors emerge when two micro-architectural features interact: (i) concurrency, where multiple CPU cores simultaneously execute parts of a program, and (ii) out-of-order execution, where a CPU core reorders instructions to increase throughput. Programs can non-deterministically show one of various weak memory behaviors, meaning it could behave differently when executing again. Those behaviors differ between architectures. When migrating programs from x86 to Arm or RISC-V, the same program could non-deterministically show behaviors never observed on x86.
In part I, we look at binary translators, which are software systems that translate compiled binary programs between architectures. We study the translation process of three such real-world systems, identify errors in their translation of concurrency primitives, and fix them. We propose mathematically-rigorous weak memory models for these translators. We then define mapping schemes to translate concurrency primitives one-by-one from x86 to Arm and RISC-V. With the formal semantics, we prove those mapping schemes correct in the Agda proof assistant.
In part II, we study the common structure of our weak memory proofs written in Agda. As those proofs are often large, complex, and rigid, we identify their common structures for which we identify domain-specific abstractions. We implement those abstractions in our novel Agda proof framework Burrow to greatly simplify writing future weak memory proofs.
In part III, we use dynamic analysis to identify weak behaviors that were never seen on x86 but could appear on Arm. Our analysis simulates the program’s execution with the formal weak memory semantics of x86 and Arm. This analysis identifies only the new behaviors the program shows in practice. After finding any new behavior on Arm, we judiciously modify the program to eliminate only that behavior.
In part I, we look at binary translators, which are software systems that translate compiled binary programs between architectures. We study the translation process of three such real-world systems, identify errors in their translation of concurrency primitives, and fix them. We propose mathematically-rigorous weak memory models for these translators. We then define mapping schemes to translate concurrency primitives one-by-one from x86 to Arm and RISC-V. With the formal semantics, we prove those mapping schemes correct in the Agda proof assistant.
In part II, we study the common structure of our weak memory proofs written in Agda. As those proofs are often large, complex, and rigid, we identify their common structures for which we identify domain-specific abstractions. We implement those abstractions in our novel Agda proof framework Burrow to greatly simplify writing future weak memory proofs.
In part III, we use dynamic analysis to identify weak behaviors that were never seen on x86 but could appear on Arm. Our analysis simulates the program’s execution with the formal weak memory semantics of x86 and Arm. This analysis identifies only the new behaviors the program shows in practice. After finding any new behavior on Arm, we judiciously modify the program to eliminate only that behavior.
Sunlight-based Passive VLC
Utilizing the sun to establish wireless connections
Nowadays,wireless connectivity is ubiquitous: humans use smartphones, smartwatches, laptops and other devices, while at the same time, the Internet of Things (IoT) is adding millions of connected objects. This large number of devices uses mainly the radio frequency (RF) spectrumfor communication. And a direct consequence of this exponential growth is the scarcity of free RF bands to cope with this demand.
To tackle this challenge, researchers have proposed using a different carrier: visible light. With Visible Light Communications (VLC), devices communicate with each other by modulating the intensity of their light-emitting diodes (LEDs) and demodulating it using light sensors. The key advantage of VLC is the utilization of the visible light spectrum, with free bands that do not interfere with traditional RF systems. Nonetheless, despite the efficiency of LED technology, luminaries still require several Watts to generate light. The need for this considerable amount of energy has triggered interest in a new research area: Passive VLC. The fundamental principle of Passive VLC is to exploit ambient light to create wireless links, thus reducing the energy required by transmitters to generate their own light.
Passive VLC is a promising area, but poses a daring challenge: modulate light without any control over the source. The research community has proposed using optical surfaces that block or reflect light dynamically as modulators, but these platforms provide limited data rates, ranging froma few tens of bps to a few kbps. Moreover, using the sun as the source of ambient light introduces another challenge: variations in position and intensity.
This dissertation aims to improve the performance of Passive VLC systems operating with sunlight, with a particular focus on increasing the data rate and resilience to the changing sun’s position.
Our first contribution is a short-range wireless link using a tiny screen as a transmitter and a camera as a receiver. The screen is a reflective surface, adapted to work with ambient light. The sunlight reaching the screen is modulated to transmit information to a smartphone’s camera, creating a stream of optical data. This screen-to-camera link using sunlight attains up to 10 kbps, ten times faster than previous similar systems, working from sunrise to sunset - independent of the sun’s position.
Inspired by the concept of Li-Fi, which combines illumination and VLC, our second contribution envisions the creation of a natural light bulb with wireless communication capabilities. Our design combines optical modulators, optical filters and sunlight collectors to track the sun’s position during the day and radiate modulated beams of sunlight in indoor scenarios. These beams of natural light provide illumination and communication and are the first to divide sunlight into two color channels to double the data rate.
Our third contribution proposes a novel link for robots to communicate using sunlight. We leverage a material used in solar technology, the Luminescent Solar Concentrator (LSC). An LSC surface absorbs light fromits top and emits it on its edges. We place LSCs on top of robots, together with liquid crystal cells (LCs), so sunlight arriving from the top can be modulated into data packets transmitted toward the edges. This novel communication systemallows task coordination between robots using sunlight.
Overall, this dissertation presents new Passive VLC systems focusing on applications that exploit the sun as the light source. Within this scenario, our focus has been to increase the data rate, with the first two contributions, and on making the systems resilient to the sun’s position, with all three contributions. ...
To tackle this challenge, researchers have proposed using a different carrier: visible light. With Visible Light Communications (VLC), devices communicate with each other by modulating the intensity of their light-emitting diodes (LEDs) and demodulating it using light sensors. The key advantage of VLC is the utilization of the visible light spectrum, with free bands that do not interfere with traditional RF systems. Nonetheless, despite the efficiency of LED technology, luminaries still require several Watts to generate light. The need for this considerable amount of energy has triggered interest in a new research area: Passive VLC. The fundamental principle of Passive VLC is to exploit ambient light to create wireless links, thus reducing the energy required by transmitters to generate their own light.
Passive VLC is a promising area, but poses a daring challenge: modulate light without any control over the source. The research community has proposed using optical surfaces that block or reflect light dynamically as modulators, but these platforms provide limited data rates, ranging froma few tens of bps to a few kbps. Moreover, using the sun as the source of ambient light introduces another challenge: variations in position and intensity.
This dissertation aims to improve the performance of Passive VLC systems operating with sunlight, with a particular focus on increasing the data rate and resilience to the changing sun’s position.
Our first contribution is a short-range wireless link using a tiny screen as a transmitter and a camera as a receiver. The screen is a reflective surface, adapted to work with ambient light. The sunlight reaching the screen is modulated to transmit information to a smartphone’s camera, creating a stream of optical data. This screen-to-camera link using sunlight attains up to 10 kbps, ten times faster than previous similar systems, working from sunrise to sunset - independent of the sun’s position.
Inspired by the concept of Li-Fi, which combines illumination and VLC, our second contribution envisions the creation of a natural light bulb with wireless communication capabilities. Our design combines optical modulators, optical filters and sunlight collectors to track the sun’s position during the day and radiate modulated beams of sunlight in indoor scenarios. These beams of natural light provide illumination and communication and are the first to divide sunlight into two color channels to double the data rate.
Our third contribution proposes a novel link for robots to communicate using sunlight. We leverage a material used in solar technology, the Luminescent Solar Concentrator (LSC). An LSC surface absorbs light fromits top and emits it on its edges. We place LSCs on top of robots, together with liquid crystal cells (LCs), so sunlight arriving from the top can be modulated into data packets transmitted toward the edges. This novel communication systemallows task coordination between robots using sunlight.
Overall, this dissertation presents new Passive VLC systems focusing on applications that exploit the sun as the light source. Within this scenario, our focus has been to increase the data rate, with the first two contributions, and on making the systems resilient to the sun’s position, with all three contributions. ...
Nowadays,wireless connectivity is ubiquitous: humans use smartphones, smartwatches, laptops and other devices, while at the same time, the Internet of Things (IoT) is adding millions of connected objects. This large number of devices uses mainly the radio frequency (RF) spectrumfor communication. And a direct consequence of this exponential growth is the scarcity of free RF bands to cope with this demand.
To tackle this challenge, researchers have proposed using a different carrier: visible light. With Visible Light Communications (VLC), devices communicate with each other by modulating the intensity of their light-emitting diodes (LEDs) and demodulating it using light sensors. The key advantage of VLC is the utilization of the visible light spectrum, with free bands that do not interfere with traditional RF systems. Nonetheless, despite the efficiency of LED technology, luminaries still require several Watts to generate light. The need for this considerable amount of energy has triggered interest in a new research area: Passive VLC. The fundamental principle of Passive VLC is to exploit ambient light to create wireless links, thus reducing the energy required by transmitters to generate their own light.
Passive VLC is a promising area, but poses a daring challenge: modulate light without any control over the source. The research community has proposed using optical surfaces that block or reflect light dynamically as modulators, but these platforms provide limited data rates, ranging froma few tens of bps to a few kbps. Moreover, using the sun as the source of ambient light introduces another challenge: variations in position and intensity.
This dissertation aims to improve the performance of Passive VLC systems operating with sunlight, with a particular focus on increasing the data rate and resilience to the changing sun’s position.
Our first contribution is a short-range wireless link using a tiny screen as a transmitter and a camera as a receiver. The screen is a reflective surface, adapted to work with ambient light. The sunlight reaching the screen is modulated to transmit information to a smartphone’s camera, creating a stream of optical data. This screen-to-camera link using sunlight attains up to 10 kbps, ten times faster than previous similar systems, working from sunrise to sunset - independent of the sun’s position.
Inspired by the concept of Li-Fi, which combines illumination and VLC, our second contribution envisions the creation of a natural light bulb with wireless communication capabilities. Our design combines optical modulators, optical filters and sunlight collectors to track the sun’s position during the day and radiate modulated beams of sunlight in indoor scenarios. These beams of natural light provide illumination and communication and are the first to divide sunlight into two color channels to double the data rate.
Our third contribution proposes a novel link for robots to communicate using sunlight. We leverage a material used in solar technology, the Luminescent Solar Concentrator (LSC). An LSC surface absorbs light fromits top and emits it on its edges. We place LSCs on top of robots, together with liquid crystal cells (LCs), so sunlight arriving from the top can be modulated into data packets transmitted toward the edges. This novel communication systemallows task coordination between robots using sunlight.
Overall, this dissertation presents new Passive VLC systems focusing on applications that exploit the sun as the light source. Within this scenario, our focus has been to increase the data rate, with the first two contributions, and on making the systems resilient to the sun’s position, with all three contributions.
To tackle this challenge, researchers have proposed using a different carrier: visible light. With Visible Light Communications (VLC), devices communicate with each other by modulating the intensity of their light-emitting diodes (LEDs) and demodulating it using light sensors. The key advantage of VLC is the utilization of the visible light spectrum, with free bands that do not interfere with traditional RF systems. Nonetheless, despite the efficiency of LED technology, luminaries still require several Watts to generate light. The need for this considerable amount of energy has triggered interest in a new research area: Passive VLC. The fundamental principle of Passive VLC is to exploit ambient light to create wireless links, thus reducing the energy required by transmitters to generate their own light.
Passive VLC is a promising area, but poses a daring challenge: modulate light without any control over the source. The research community has proposed using optical surfaces that block or reflect light dynamically as modulators, but these platforms provide limited data rates, ranging froma few tens of bps to a few kbps. Moreover, using the sun as the source of ambient light introduces another challenge: variations in position and intensity.
This dissertation aims to improve the performance of Passive VLC systems operating with sunlight, with a particular focus on increasing the data rate and resilience to the changing sun’s position.
Our first contribution is a short-range wireless link using a tiny screen as a transmitter and a camera as a receiver. The screen is a reflective surface, adapted to work with ambient light. The sunlight reaching the screen is modulated to transmit information to a smartphone’s camera, creating a stream of optical data. This screen-to-camera link using sunlight attains up to 10 kbps, ten times faster than previous similar systems, working from sunrise to sunset - independent of the sun’s position.
Inspired by the concept of Li-Fi, which combines illumination and VLC, our second contribution envisions the creation of a natural light bulb with wireless communication capabilities. Our design combines optical modulators, optical filters and sunlight collectors to track the sun’s position during the day and radiate modulated beams of sunlight in indoor scenarios. These beams of natural light provide illumination and communication and are the first to divide sunlight into two color channels to double the data rate.
Our third contribution proposes a novel link for robots to communicate using sunlight. We leverage a material used in solar technology, the Luminescent Solar Concentrator (LSC). An LSC surface absorbs light fromits top and emits it on its edges. We place LSCs on top of robots, together with liquid crystal cells (LCs), so sunlight arriving from the top can be modulated into data packets transmitted toward the edges. This novel communication systemallows task coordination between robots using sunlight.
Overall, this dissertation presents new Passive VLC systems focusing on applications that exploit the sun as the light source. Within this scenario, our focus has been to increase the data rate, with the first two contributions, and on making the systems resilient to the sun’s position, with all three contributions.
AI/ML-based approaches are at the forefront of resource management in modern communication networks. Deep learning, in particular, enables fast and high-performing decision-making when sufficient representative training data is available to build accurate offline models. Conversely, online learning solutions operate without prior training and make decisions based on real-time observations; however, they tend to be overly conservative to ensure robustness (i.e., worst-case guarantees).
This thesis advocates optimistic learning as a decision-making framework for resource management in networked systems. An optimistic learning algorithm integrates untrusted predictions and assesses their accuracy at runtime. When predictions are accurate, these algorithms achieve performance levels comparable to offline-trained models. Crucially, they maintain the robustness of regular online learning, ensuring reliability even when predictions are inaccurate.
We focus on caching networks and propose new optimistic learning algorithms for coded caching, and whole-file caching. These algorithms provably converge to the best fixed caching allocation at an order-optimal rate, independent of prediction accuracy. However, when predictions are accurate, convergence is highly accelerated, achieving the “optimistic" premise.
We then extend our focus to scenarios where the optimization target itself changes over time. In caching, this translates to competing against dynamic caching configurations rather than a single best fixed allocation. We demonstrate that optimism is even more valuable in this setting; accurate predictions help the learner efficiently track moving targets, adapting in real-time without excessive conservatism. Furthermore, we explore the role of predictions in stateful systems, where past decisions influence future costs. In such environments, optimistic learning benefits from horizon-based predictions, leveraging forecasts over extended time windows rather than immediate next-cost predictions.
All proposed algorithms are rigorously analyzed and come with provable performance guarantees under carefully designed and explicitly stated metrics. By integrating optimistic learning into network optimization, this thesis explores the spectrum between prediction-driven and robust approaches, offering a principled framework for leveraging untrusted ML predictions in network resource allocation.
...
This thesis advocates optimistic learning as a decision-making framework for resource management in networked systems. An optimistic learning algorithm integrates untrusted predictions and assesses their accuracy at runtime. When predictions are accurate, these algorithms achieve performance levels comparable to offline-trained models. Crucially, they maintain the robustness of regular online learning, ensuring reliability even when predictions are inaccurate.
We focus on caching networks and propose new optimistic learning algorithms for coded caching, and whole-file caching. These algorithms provably converge to the best fixed caching allocation at an order-optimal rate, independent of prediction accuracy. However, when predictions are accurate, convergence is highly accelerated, achieving the “optimistic" premise.
We then extend our focus to scenarios where the optimization target itself changes over time. In caching, this translates to competing against dynamic caching configurations rather than a single best fixed allocation. We demonstrate that optimism is even more valuable in this setting; accurate predictions help the learner efficiently track moving targets, adapting in real-time without excessive conservatism. Furthermore, we explore the role of predictions in stateful systems, where past decisions influence future costs. In such environments, optimistic learning benefits from horizon-based predictions, leveraging forecasts over extended time windows rather than immediate next-cost predictions.
All proposed algorithms are rigorously analyzed and come with provable performance guarantees under carefully designed and explicitly stated metrics. By integrating optimistic learning into network optimization, this thesis explores the spectrum between prediction-driven and robust approaches, offering a principled framework for leveraging untrusted ML predictions in network resource allocation.
...
AI/ML-based approaches are at the forefront of resource management in modern communication networks. Deep learning, in particular, enables fast and high-performing decision-making when sufficient representative training data is available to build accurate offline models. Conversely, online learning solutions operate without prior training and make decisions based on real-time observations; however, they tend to be overly conservative to ensure robustness (i.e., worst-case guarantees).
This thesis advocates optimistic learning as a decision-making framework for resource management in networked systems. An optimistic learning algorithm integrates untrusted predictions and assesses their accuracy at runtime. When predictions are accurate, these algorithms achieve performance levels comparable to offline-trained models. Crucially, they maintain the robustness of regular online learning, ensuring reliability even when predictions are inaccurate.
We focus on caching networks and propose new optimistic learning algorithms for coded caching, and whole-file caching. These algorithms provably converge to the best fixed caching allocation at an order-optimal rate, independent of prediction accuracy. However, when predictions are accurate, convergence is highly accelerated, achieving the “optimistic" premise.
We then extend our focus to scenarios where the optimization target itself changes over time. In caching, this translates to competing against dynamic caching configurations rather than a single best fixed allocation. We demonstrate that optimism is even more valuable in this setting; accurate predictions help the learner efficiently track moving targets, adapting in real-time without excessive conservatism. Furthermore, we explore the role of predictions in stateful systems, where past decisions influence future costs. In such environments, optimistic learning benefits from horizon-based predictions, leveraging forecasts over extended time windows rather than immediate next-cost predictions.
All proposed algorithms are rigorously analyzed and come with provable performance guarantees under carefully designed and explicitly stated metrics. By integrating optimistic learning into network optimization, this thesis explores the spectrum between prediction-driven and robust approaches, offering a principled framework for leveraging untrusted ML predictions in network resource allocation.
This thesis advocates optimistic learning as a decision-making framework for resource management in networked systems. An optimistic learning algorithm integrates untrusted predictions and assesses their accuracy at runtime. When predictions are accurate, these algorithms achieve performance levels comparable to offline-trained models. Crucially, they maintain the robustness of regular online learning, ensuring reliability even when predictions are inaccurate.
We focus on caching networks and propose new optimistic learning algorithms for coded caching, and whole-file caching. These algorithms provably converge to the best fixed caching allocation at an order-optimal rate, independent of prediction accuracy. However, when predictions are accurate, convergence is highly accelerated, achieving the “optimistic" premise.
We then extend our focus to scenarios where the optimization target itself changes over time. In caching, this translates to competing against dynamic caching configurations rather than a single best fixed allocation. We demonstrate that optimism is even more valuable in this setting; accurate predictions help the learner efficiently track moving targets, adapting in real-time without excessive conservatism. Furthermore, we explore the role of predictions in stateful systems, where past decisions influence future costs. In such environments, optimistic learning benefits from horizon-based predictions, leveraging forecasts over extended time windows rather than immediate next-cost predictions.
All proposed algorithms are rigorously analyzed and come with provable performance guarantees under carefully designed and explicitly stated metrics. By integrating optimistic learning into network optimization, this thesis explores the spectrum between prediction-driven and robust approaches, offering a principled framework for leveraging untrusted ML predictions in network resource allocation.
Embedded real-time systems, essential in industries like automotive, aviation, and medical devices, increasingly rely on multi-core platforms for efficient parallel processing to meet rising computational demands. However, they face significant challenges with energy consumption, particularly in energy-constrained environments where high energy usage affects system reliability and longevity. Consequently, researchers have focused on enhancing energy efficiency by dynamically adjusting the energy consumption profiles of system components to align with workload demands by leveraging energy management techniques such as dynamic voltage and frequency scaling (DVFS) while meeting timing requirements.
In this work, we consider the problem of energy-aware speed assignment for a hard real-time workload scheduled with an online global non-preemptive work-conserving job-level-fixed-priority scheduler on a multi-core platform with a discrete core-level DVFS model. Our solution determines a speed for each job in the workload to reduce the energy consumption of the system while ensuring the timing requirements of the workload. As we consider a non-preemptive execution model, the timing uncertainties in the workload can result in scheduling anomalies. Therefore, we use the schedule abstraction graph (SAG), a reachability-based response time analysis tool to explore all possible execution scenarios and identify any potential deadline violations.
The key idea of our work is to iteratively explore all execution scenarios using SAG with all jobs running at the most energy-efficient speeds while readjusting the speeds to resolve any potential deadline violation. To limit the speed readjustment search space, we present a novel approach for identifying connections between jobs scheduled on a multi-core platform, considering all possible speed combinations with the selected speed settings.
Our evaluations show that our solution can reduce the energy consumption of a system by 25.85%, on average with an average runtime overhead of 7.7 times higher than that of the schedulability analysis when running all jobs at the highest speed, for the speed range {0.74, 0.80, 0.87, 0.94, 1.00}. This demonstrates a considerable energy reduction potential with energy-aware speed assignment for static slack reclamation using a schedule abstraction graph. ...
In this work, we consider the problem of energy-aware speed assignment for a hard real-time workload scheduled with an online global non-preemptive work-conserving job-level-fixed-priority scheduler on a multi-core platform with a discrete core-level DVFS model. Our solution determines a speed for each job in the workload to reduce the energy consumption of the system while ensuring the timing requirements of the workload. As we consider a non-preemptive execution model, the timing uncertainties in the workload can result in scheduling anomalies. Therefore, we use the schedule abstraction graph (SAG), a reachability-based response time analysis tool to explore all possible execution scenarios and identify any potential deadline violations.
The key idea of our work is to iteratively explore all execution scenarios using SAG with all jobs running at the most energy-efficient speeds while readjusting the speeds to resolve any potential deadline violation. To limit the speed readjustment search space, we present a novel approach for identifying connections between jobs scheduled on a multi-core platform, considering all possible speed combinations with the selected speed settings.
Our evaluations show that our solution can reduce the energy consumption of a system by 25.85%, on average with an average runtime overhead of 7.7 times higher than that of the schedulability analysis when running all jobs at the highest speed, for the speed range {0.74, 0.80, 0.87, 0.94, 1.00}. This demonstrates a considerable energy reduction potential with energy-aware speed assignment for static slack reclamation using a schedule abstraction graph. ...
Embedded real-time systems, essential in industries like automotive, aviation, and medical devices, increasingly rely on multi-core platforms for efficient parallel processing to meet rising computational demands. However, they face significant challenges with energy consumption, particularly in energy-constrained environments where high energy usage affects system reliability and longevity. Consequently, researchers have focused on enhancing energy efficiency by dynamically adjusting the energy consumption profiles of system components to align with workload demands by leveraging energy management techniques such as dynamic voltage and frequency scaling (DVFS) while meeting timing requirements.
In this work, we consider the problem of energy-aware speed assignment for a hard real-time workload scheduled with an online global non-preemptive work-conserving job-level-fixed-priority scheduler on a multi-core platform with a discrete core-level DVFS model. Our solution determines a speed for each job in the workload to reduce the energy consumption of the system while ensuring the timing requirements of the workload. As we consider a non-preemptive execution model, the timing uncertainties in the workload can result in scheduling anomalies. Therefore, we use the schedule abstraction graph (SAG), a reachability-based response time analysis tool to explore all possible execution scenarios and identify any potential deadline violations.
The key idea of our work is to iteratively explore all execution scenarios using SAG with all jobs running at the most energy-efficient speeds while readjusting the speeds to resolve any potential deadline violation. To limit the speed readjustment search space, we present a novel approach for identifying connections between jobs scheduled on a multi-core platform, considering all possible speed combinations with the selected speed settings.
Our evaluations show that our solution can reduce the energy consumption of a system by 25.85%, on average with an average runtime overhead of 7.7 times higher than that of the schedulability analysis when running all jobs at the highest speed, for the speed range {0.74, 0.80, 0.87, 0.94, 1.00}. This demonstrates a considerable energy reduction potential with energy-aware speed assignment for static slack reclamation using a schedule abstraction graph.
In this work, we consider the problem of energy-aware speed assignment for a hard real-time workload scheduled with an online global non-preemptive work-conserving job-level-fixed-priority scheduler on a multi-core platform with a discrete core-level DVFS model. Our solution determines a speed for each job in the workload to reduce the energy consumption of the system while ensuring the timing requirements of the workload. As we consider a non-preemptive execution model, the timing uncertainties in the workload can result in scheduling anomalies. Therefore, we use the schedule abstraction graph (SAG), a reachability-based response time analysis tool to explore all possible execution scenarios and identify any potential deadline violations.
The key idea of our work is to iteratively explore all execution scenarios using SAG with all jobs running at the most energy-efficient speeds while readjusting the speeds to resolve any potential deadline violation. To limit the speed readjustment search space, we present a novel approach for identifying connections between jobs scheduled on a multi-core platform, considering all possible speed combinations with the selected speed settings.
Our evaluations show that our solution can reduce the energy consumption of a system by 25.85%, on average with an average runtime overhead of 7.7 times higher than that of the schedulability analysis when running all jobs at the highest speed, for the speed range {0.74, 0.80, 0.87, 0.94, 1.00}. This demonstrates a considerable energy reduction potential with energy-aware speed assignment for static slack reclamation using a schedule abstraction graph.
In practical situations, computer vision technique is applied to solve various tasks, including image classification, object detection, image segmentation, and so on. The commonly used supervised learning training paradigm for the network models used to solve these tasks requires training data as well as the ground truth labels specifying the data samples' reference information for the task. However, getting labels for every task would be expensive or even almost impossible, such as medical images due to privacy reasons and expert annotations from medical professionals and facial recognition also because of privacy concerns. Many large-scale general datasets exist, like ILSVRC2012 for both image classification and object detection tasks and COCO also for objection tasks. , and the corresponding pre-trained models whose knowledge can be transferred to other fields. While deeper models typically have better performance and can learn better feature representation from the same tasks, increasing network models introduces difficulties in practical deployment, especially regarding resource limitation and response latency. We hope to explore the learning methods in knowledge distillation to help the smaller student network learn better features from the unlabeled training data and have better transfer performance on downstream tasks. With the remarkable success of contrastive learning, it has become one of the most promising methods of learning from unlabeled data. In this thesis, we proposed an unsupervised knowledge distillation method that applies a contrastive learning method to construct and extract relational knowledge from the feature representations of the intermediate layers as well as the final layer. The evaluation with the ILSVRC2012 dataset proves the effectiveness of the proposed method on feature learning as the method helped the ResNet-18 model achieve a 2% improvement in linear evaluation accuracy compared to the baseline model. Extensive experiments were conducted on eight transfer learning tasks, and the model trained by the proposed method outperformed its baseline model in all the eight classification tasks and also achieved better fine-tuning accuracy in the situation where only a small fraction of the ground truth label was available for fine-tuning.
...
...
In practical situations, computer vision technique is applied to solve various tasks, including image classification, object detection, image segmentation, and so on. The commonly used supervised learning training paradigm for the network models used to solve these tasks requires training data as well as the ground truth labels specifying the data samples' reference information for the task. However, getting labels for every task would be expensive or even almost impossible, such as medical images due to privacy reasons and expert annotations from medical professionals and facial recognition also because of privacy concerns. Many large-scale general datasets exist, like ILSVRC2012 for both image classification and object detection tasks and COCO also for objection tasks. , and the corresponding pre-trained models whose knowledge can be transferred to other fields. While deeper models typically have better performance and can learn better feature representation from the same tasks, increasing network models introduces difficulties in practical deployment, especially regarding resource limitation and response latency. We hope to explore the learning methods in knowledge distillation to help the smaller student network learn better features from the unlabeled training data and have better transfer performance on downstream tasks. With the remarkable success of contrastive learning, it has become one of the most promising methods of learning from unlabeled data. In this thesis, we proposed an unsupervised knowledge distillation method that applies a contrastive learning method to construct and extract relational knowledge from the feature representations of the intermediate layers as well as the final layer. The evaluation with the ILSVRC2012 dataset proves the effectiveness of the proposed method on feature learning as the method helped the ResNet-18 model achieve a 2% improvement in linear evaluation accuracy compared to the baseline model. Extensive experiments were conducted on eight transfer learning tasks, and the model trained by the proposed method outperformed its baseline model in all the eight classification tasks and also achieved better fine-tuning accuracy in the situation where only a small fraction of the ground truth label was available for fine-tuning.
The proliferation of video recording devices and facial recognition technology has led to significant privacy concerns, as surveillance systems can capture and identify individuals without their consent. Traditional facial obfuscation systems, which introduce pixel-level perturbations to images, aim to protect privacy by preventing unauthorized facial recognition. However, these systems are vulnerable to inversion attacks, where attackers can reverse the perturbations to restore original images, compromising privacy. This thesis addresses these vulnerabilities by proposing HyperObf, a novel approach utilizing HyperNet technology to generate unique obfuscation networks for each user. HyperObf ensures that each user’s images are distinctly protected, making it challenging for attackers to reverse-engineer the obfuscations. Our experiments demonstrate that inversion attacks can significantly degrade the protection offered by static obfuscation systems, with restored images achieving face recognition accuracy close to that of original images.
In contrast, HyperObf effectively mitigates these attacks, reducing the attack success rate to 30% compared to 60% for existing methods. Additionally, HyperObf can generate 100 personalized MaskNets in 0.2 seconds using high-performance computing resources. These findings highlight the potential of HyperObf to enhance privacy protection against unauthorized facial recognition and inversion attacks in the digital age. ...
In contrast, HyperObf effectively mitigates these attacks, reducing the attack success rate to 30% compared to 60% for existing methods. Additionally, HyperObf can generate 100 personalized MaskNets in 0.2 seconds using high-performance computing resources. These findings highlight the potential of HyperObf to enhance privacy protection against unauthorized facial recognition and inversion attacks in the digital age. ...
The proliferation of video recording devices and facial recognition technology has led to significant privacy concerns, as surveillance systems can capture and identify individuals without their consent. Traditional facial obfuscation systems, which introduce pixel-level perturbations to images, aim to protect privacy by preventing unauthorized facial recognition. However, these systems are vulnerable to inversion attacks, where attackers can reverse the perturbations to restore original images, compromising privacy. This thesis addresses these vulnerabilities by proposing HyperObf, a novel approach utilizing HyperNet technology to generate unique obfuscation networks for each user. HyperObf ensures that each user’s images are distinctly protected, making it challenging for attackers to reverse-engineer the obfuscations. Our experiments demonstrate that inversion attacks can significantly degrade the protection offered by static obfuscation systems, with restored images achieving face recognition accuracy close to that of original images.
In contrast, HyperObf effectively mitigates these attacks, reducing the attack success rate to 30% compared to 60% for existing methods. Additionally, HyperObf can generate 100 personalized MaskNets in 0.2 seconds using high-performance computing resources. These findings highlight the potential of HyperObf to enhance privacy protection against unauthorized facial recognition and inversion attacks in the digital age.
In contrast, HyperObf effectively mitigates these attacks, reducing the attack success rate to 30% compared to 60% for existing methods. Additionally, HyperObf can generate 100 personalized MaskNets in 0.2 seconds using high-performance computing resources. These findings highlight the potential of HyperObf to enhance privacy protection against unauthorized facial recognition and inversion attacks in the digital age.
Path finding is an important component in solving a wide array of engineering problems, ranging from video games to real-life applications such as automated warehouse management and autonomous vehicles.
Path finding algorithms are designed to solve complex problems, and in order to do so, assumptions are necessary to simplify the problems.
While these assumptions are important, using them makes the obtained algorithms less applicable to a real-life scenario, and as such, verifying how lifting some of them would affect the obtained results is worth pursuing.
Two main assumptions were identified and subsequently lifted.
First, classic multi-agent path finding algorithms use a centralized approach, where solutions are computed before execution.
This results in a long computation period followed by execution. Lifting this assumption results in a decentralized approach where agents solve conflicts on the go, while approaching their target.
The second assumption made by state of the art algorithms is that agents participating in a multi-agent path finding problem share a common goal: minimizing a global cost function.
This is not always applicable, as in a real-life scenario participating agents can have selfish goals.
This assumptions has been lifted by allowing agents to negotiate their paths by trading with the other participants to create better solutions for themselves.
The Selfish Localized Pathfinding (SLP) algorithm has been designed to lift these assumptions. It describes a decentralized algorithm that allows participating agents to negotiate their paths, which makes a good candidate for an application closer to real-life.
The SLP algorithm has been tested in order to evaluate its performance, both in terms of its ability to solve a set of test cases, and in terms of the cost incurred by the participating agents.
SLP performed well in varied domains.
SLP solved significantly more cases than Conflict Based Search, a centralized state of the art path finding algorithm.
This comes at the expense of an increase in the path lengths obtained by the algorithm.
This downside is offset by the significant decrease in the time required to solve problems, which can be divided in small clusters due to the decentralized approach of the algorithm.
On the whole, SLP can provide a good alternative to Conflict Based Search. ...
Path finding algorithms are designed to solve complex problems, and in order to do so, assumptions are necessary to simplify the problems.
While these assumptions are important, using them makes the obtained algorithms less applicable to a real-life scenario, and as such, verifying how lifting some of them would affect the obtained results is worth pursuing.
Two main assumptions were identified and subsequently lifted.
First, classic multi-agent path finding algorithms use a centralized approach, where solutions are computed before execution.
This results in a long computation period followed by execution. Lifting this assumption results in a decentralized approach where agents solve conflicts on the go, while approaching their target.
The second assumption made by state of the art algorithms is that agents participating in a multi-agent path finding problem share a common goal: minimizing a global cost function.
This is not always applicable, as in a real-life scenario participating agents can have selfish goals.
This assumptions has been lifted by allowing agents to negotiate their paths by trading with the other participants to create better solutions for themselves.
The Selfish Localized Pathfinding (SLP) algorithm has been designed to lift these assumptions. It describes a decentralized algorithm that allows participating agents to negotiate their paths, which makes a good candidate for an application closer to real-life.
The SLP algorithm has been tested in order to evaluate its performance, both in terms of its ability to solve a set of test cases, and in terms of the cost incurred by the participating agents.
SLP performed well in varied domains.
SLP solved significantly more cases than Conflict Based Search, a centralized state of the art path finding algorithm.
This comes at the expense of an increase in the path lengths obtained by the algorithm.
This downside is offset by the significant decrease in the time required to solve problems, which can be divided in small clusters due to the decentralized approach of the algorithm.
On the whole, SLP can provide a good alternative to Conflict Based Search. ...
Path finding is an important component in solving a wide array of engineering problems, ranging from video games to real-life applications such as automated warehouse management and autonomous vehicles.
Path finding algorithms are designed to solve complex problems, and in order to do so, assumptions are necessary to simplify the problems.
While these assumptions are important, using them makes the obtained algorithms less applicable to a real-life scenario, and as such, verifying how lifting some of them would affect the obtained results is worth pursuing.
Two main assumptions were identified and subsequently lifted.
First, classic multi-agent path finding algorithms use a centralized approach, where solutions are computed before execution.
This results in a long computation period followed by execution. Lifting this assumption results in a decentralized approach where agents solve conflicts on the go, while approaching their target.
The second assumption made by state of the art algorithms is that agents participating in a multi-agent path finding problem share a common goal: minimizing a global cost function.
This is not always applicable, as in a real-life scenario participating agents can have selfish goals.
This assumptions has been lifted by allowing agents to negotiate their paths by trading with the other participants to create better solutions for themselves.
The Selfish Localized Pathfinding (SLP) algorithm has been designed to lift these assumptions. It describes a decentralized algorithm that allows participating agents to negotiate their paths, which makes a good candidate for an application closer to real-life.
The SLP algorithm has been tested in order to evaluate its performance, both in terms of its ability to solve a set of test cases, and in terms of the cost incurred by the participating agents.
SLP performed well in varied domains.
SLP solved significantly more cases than Conflict Based Search, a centralized state of the art path finding algorithm.
This comes at the expense of an increase in the path lengths obtained by the algorithm.
This downside is offset by the significant decrease in the time required to solve problems, which can be divided in small clusters due to the decentralized approach of the algorithm.
On the whole, SLP can provide a good alternative to Conflict Based Search.
Path finding algorithms are designed to solve complex problems, and in order to do so, assumptions are necessary to simplify the problems.
While these assumptions are important, using them makes the obtained algorithms less applicable to a real-life scenario, and as such, verifying how lifting some of them would affect the obtained results is worth pursuing.
Two main assumptions were identified and subsequently lifted.
First, classic multi-agent path finding algorithms use a centralized approach, where solutions are computed before execution.
This results in a long computation period followed by execution. Lifting this assumption results in a decentralized approach where agents solve conflicts on the go, while approaching their target.
The second assumption made by state of the art algorithms is that agents participating in a multi-agent path finding problem share a common goal: minimizing a global cost function.
This is not always applicable, as in a real-life scenario participating agents can have selfish goals.
This assumptions has been lifted by allowing agents to negotiate their paths by trading with the other participants to create better solutions for themselves.
The Selfish Localized Pathfinding (SLP) algorithm has been designed to lift these assumptions. It describes a decentralized algorithm that allows participating agents to negotiate their paths, which makes a good candidate for an application closer to real-life.
The SLP algorithm has been tested in order to evaluate its performance, both in terms of its ability to solve a set of test cases, and in terms of the cost incurred by the participating agents.
SLP performed well in varied domains.
SLP solved significantly more cases than Conflict Based Search, a centralized state of the art path finding algorithm.
This comes at the expense of an increase in the path lengths obtained by the algorithm.
This downside is offset by the significant decrease in the time required to solve problems, which can be divided in small clusters due to the decentralized approach of the algorithm.
On the whole, SLP can provide a good alternative to Conflict Based Search.
Human activity recognition plays an interesting and important role nowadays as there are a variety of use cases. It is utilized in health monitoring, in the development of human-computer interaction system and in security monitoring. However current methods involve usage of privacy sensitive data and impractical sensors for everyday usage. To tackle this problem, we aim to answer the research question "How to maximize the capabilities of in-mouth sensors for human activity recognition?". The main contributions of this paper are the classification of different gestures using an in-mouth device, implementation of a classifier directly onto a microcontroller and the evaluation whether the models can generalize to multiple people. To investigate this, we experimented with popular classical machine learning classifiers: Decision Tree, K-Nearest Neighbors, Support Vector Machine, Logistic Regression and Random Forest classifiers. The results shows that the F1-score of all classification problems are above 80% using the various classifiers along with different parameters.
...
Human activity recognition plays an interesting and important role nowadays as there are a variety of use cases. It is utilized in health monitoring, in the development of human-computer interaction system and in security monitoring. However current methods involve usage of privacy sensitive data and impractical sensors for everyday usage. To tackle this problem, we aim to answer the research question "How to maximize the capabilities of in-mouth sensors for human activity recognition?". The main contributions of this paper are the classification of different gestures using an in-mouth device, implementation of a classifier directly onto a microcontroller and the evaluation whether the models can generalize to multiple people. To investigate this, we experimented with popular classical machine learning classifiers: Decision Tree, K-Nearest Neighbors, Support Vector Machine, Logistic Regression and Random Forest classifiers. The results shows that the F1-score of all classification problems are above 80% using the various classifiers along with different parameters.
Identifying Speaking and Drinking Events Within Audio Recordings for Multiactivity Analysis
Rethinking Ubiquitous Smart Sensing of Social Behaviour in the Wild
Multiactivity analysis investigates one's coordination of actions within a social context, such as gestures and speech, usually using video recordings of the social activity, to further understand the rules of human behaviour. This paper focuses specifically on the coordination between speaking and drinking activities within a social setting, and explores the possibility of automatically identifying these events using audio captured from a drinking glass. As social interactions occur in vastly different contexts, this paper also investigates the effect that background noise might have on the accuracy of identifying these events. Different parameters and audio features were compared. Linear classification models LR and SVM with a linear kernel were able to achieve 100% accuracy for all sample lengths between 2 and 8 seconds using the first 20 PCA components from 60 audio features. The best performing feature in identifying speaking and drinking events was MFCCs, achieving an F1 score of 99.4% on average across models with a training sample length of 3 seconds. Background noise had different effects on classification accuracy depending on the type, with music lowering the F1 score to 74.3%, noisy room audio to 64.7%, and podcast audio simulating the presence of other speakers to 59.6% using MFCCs and a 3-second sample length
...
Multiactivity analysis investigates one's coordination of actions within a social context, such as gestures and speech, usually using video recordings of the social activity, to further understand the rules of human behaviour. This paper focuses specifically on the coordination between speaking and drinking activities within a social setting, and explores the possibility of automatically identifying these events using audio captured from a drinking glass. As social interactions occur in vastly different contexts, this paper also investigates the effect that background noise might have on the accuracy of identifying these events. Different parameters and audio features were compared. Linear classification models LR and SVM with a linear kernel were able to achieve 100% accuracy for all sample lengths between 2 and 8 seconds using the first 20 PCA components from 60 audio features. The best performing feature in identifying speaking and drinking events was MFCCs, achieving an F1 score of 99.4% on average across models with a training sample length of 3 seconds. Background noise had different effects on classification accuracy depending on the type, with music lowering the F1 score to 74.3%, noisy room audio to 64.7%, and podcast audio simulating the presence of other speakers to 59.6% using MFCCs and a 3-second sample length
Personalized Gesture Range Detection Using Transductive Parameter Transfer
Rethinking Ubiquitous Smart Sensing of Social Behaviour In The Wild
This research investigates the detection of gestures using a torso-worn accelerometer sensor. Using the Conflab dataset, we focus on gestures during conversations in mingling scenarios. Due to significant variability in gesture styles among individuals, traditional methods face challenges in building personalized models. Our experiments demonstrate that Transductive Parameter Transfer (TPT), an adaptive transfer learning method, can more effectively model these individual differences in gesturing. To gain insights into individual expressiveness, we classify gestures into three classes: 'no gesture,' 'normal,' and 'large' gestures. TPT performed an average AUC score of 0.84 in binary classification and 0.77 in multiclass classification. These findings highlight the potential of using a single torso-worn accelerometer to understand social behavior in naturalistic settings.
...
This research investigates the detection of gestures using a torso-worn accelerometer sensor. Using the Conflab dataset, we focus on gestures during conversations in mingling scenarios. Due to significant variability in gesture styles among individuals, traditional methods face challenges in building personalized models. Our experiments demonstrate that Transductive Parameter Transfer (TPT), an adaptive transfer learning method, can more effectively model these individual differences in gesturing. To gain insights into individual expressiveness, we classify gestures into three classes: 'no gesture,' 'normal,' and 'large' gestures. TPT performed an average AUC score of 0.84 in binary classification and 0.77 in multiclass classification. These findings highlight the potential of using a single torso-worn accelerometer to understand social behavior in naturalistic settings.
Bugs in Haskell Programs
What are the different stages of bugs in Haskell programs?
Various studies already exist about the lifecycle of software programs written in languages like Java, C and C++, but this is an under-reported area for the pure, functional programming language Haskell. This report explores steps in the development of Haskell programs, and particularly, of their bugs. By gaining more knowledge about a bug’s development and its different stages, possible patterns or trends can be found. Using these insights, developers could get a better understanding of the development process, and optimise it. The main question of the research is formulated as ‘What are the different stages of bugs in Haskell programs?’. We consider three different stages: bug introduction, bug detection and bug fixing. To gain more knowledge about these stages, different open-source Haskell repositories, and their bugs were analysed. We found the median time between the bug introduction and detection is 381 days. The median time between the bug detection and fixing turned out to be 3 days. Most bugs were detected and fixed at similar rates regardless of their complexity, except for bugs that were detected by the same developer who introduced the bug. In this case, the time it takes to detect the bug is 30% of the global median. These results motivate developers to check their code more extensively, as they detect bugs quicker than other developers would. It also motivates developers to work with shared code ownership, as having more developers familiar with the code should reduce bug-detection time. Moreover, this study shows that single-statement bugs, which needed a simple fix, occur 2.5 times more often in code when no test is written, which argues for developers to write more test code. Lastly, we recommend repository owners to be more strict with developers working structured and precisely so that they follow Git’s and the repository’s standards and write test code.
...
...
Various studies already exist about the lifecycle of software programs written in languages like Java, C and C++, but this is an under-reported area for the pure, functional programming language Haskell. This report explores steps in the development of Haskell programs, and particularly, of their bugs. By gaining more knowledge about a bug’s development and its different stages, possible patterns or trends can be found. Using these insights, developers could get a better understanding of the development process, and optimise it. The main question of the research is formulated as ‘What are the different stages of bugs in Haskell programs?’. We consider three different stages: bug introduction, bug detection and bug fixing. To gain more knowledge about these stages, different open-source Haskell repositories, and their bugs were analysed. We found the median time between the bug introduction and detection is 381 days. The median time between the bug detection and fixing turned out to be 3 days. Most bugs were detected and fixed at similar rates regardless of their complexity, except for bugs that were detected by the same developer who introduced the bug. In this case, the time it takes to detect the bug is 30% of the global median. These results motivate developers to check their code more extensively, as they detect bugs quicker than other developers would. It also motivates developers to work with shared code ownership, as having more developers familiar with the code should reduce bug-detection time. Moreover, this study shows that single-statement bugs, which needed a simple fix, occur 2.5 times more often in code when no test is written, which argues for developers to write more test code. Lastly, we recommend repository owners to be more strict with developers working structured and precisely so that they follow Git’s and the repository’s standards and write test code.
Emotion Recognition in Virtual Reality
Creation and validation of a VR-based multi-modal emotion recognition dataset
Emotion recognition in Virtual Reality(VR) has the potential to offer numerous benefits across various sectors such as mental healthcare, education, marketing, entertainment, etc. Although emotion recognition itself is a mature field, the sub-field of VR-based emotion recognition is still in its early stages of development. It was found that a limiting factor in the progress of this field is a lack of sufficient data for research and development of advanced deep learning models. Also, the equipment currently used to measure emotion-related signals is expensive and impractical for general usage. This thesis aims to support the progress in this field by creating a VR-based emotion recognition dataset using VR equipment only. This addresses the problem of insufficient data available for research and development, and also reduces the reliance on expensive and impractical equipment for emotion recognition.
To create a good quality dataset, several important things had to be addressed. First of all, the stimuli to evoke the emotions had to be carefully selected to ensure that genuine emotional responses were evoked and recorded in the dataset. Then, an efficient data collection system had to be created to ensure that the data collection process ran effectively, smoothly and consistently. Then, a proper labeling process had to be designed to annotate the data as accurately as possible. Finally, the compiled dataset was validated by showing that the chosen stimuli were effective in evoking the intended emotions. This was verified through the analysis of pupil response data, which is one of the recorded data modalities. ...
To create a good quality dataset, several important things had to be addressed. First of all, the stimuli to evoke the emotions had to be carefully selected to ensure that genuine emotional responses were evoked and recorded in the dataset. Then, an efficient data collection system had to be created to ensure that the data collection process ran effectively, smoothly and consistently. Then, a proper labeling process had to be designed to annotate the data as accurately as possible. Finally, the compiled dataset was validated by showing that the chosen stimuli were effective in evoking the intended emotions. This was verified through the analysis of pupil response data, which is one of the recorded data modalities. ...
Emotion recognition in Virtual Reality(VR) has the potential to offer numerous benefits across various sectors such as mental healthcare, education, marketing, entertainment, etc. Although emotion recognition itself is a mature field, the sub-field of VR-based emotion recognition is still in its early stages of development. It was found that a limiting factor in the progress of this field is a lack of sufficient data for research and development of advanced deep learning models. Also, the equipment currently used to measure emotion-related signals is expensive and impractical for general usage. This thesis aims to support the progress in this field by creating a VR-based emotion recognition dataset using VR equipment only. This addresses the problem of insufficient data available for research and development, and also reduces the reliance on expensive and impractical equipment for emotion recognition.
To create a good quality dataset, several important things had to be addressed. First of all, the stimuli to evoke the emotions had to be carefully selected to ensure that genuine emotional responses were evoked and recorded in the dataset. Then, an efficient data collection system had to be created to ensure that the data collection process ran effectively, smoothly and consistently. Then, a proper labeling process had to be designed to annotate the data as accurately as possible. Finally, the compiled dataset was validated by showing that the chosen stimuli were effective in evoking the intended emotions. This was verified through the analysis of pupil response data, which is one of the recorded data modalities.
To create a good quality dataset, several important things had to be addressed. First of all, the stimuli to evoke the emotions had to be carefully selected to ensure that genuine emotional responses were evoked and recorded in the dataset. Then, an efficient data collection system had to be created to ensure that the data collection process ran effectively, smoothly and consistently. Then, a proper labeling process had to be designed to annotate the data as accurately as possible. Finally, the compiled dataset was validated by showing that the chosen stimuli were effective in evoking the intended emotions. This was verified through the analysis of pupil response data, which is one of the recorded data modalities.
This thesis project explores how V2X communication between electric vehicles (EV) and charging stations can be used to reduce en-route charging times. This is done using standardized V2X messages for EV charging, as well as proposing an extension to these messages to include data on the intentions of other vehicles. As individual vehicles can have significant effects on the total waiting time at EV charging stations, knowing the intentions of other vehicles will allow drivers to better avoid congested charging stations and achieve a lower total journey time. The performance of the system is evaluated using state-of-the-art traffic and communications simulators showing that using V2X can reduce journey times by 70%. Aside from demonstrating the performance using simulation the system has also been implemented on a real life test vehicle using functional V2X hardware to show that the system is viable for implementation.
...
This thesis project explores how V2X communication between electric vehicles (EV) and charging stations can be used to reduce en-route charging times. This is done using standardized V2X messages for EV charging, as well as proposing an extension to these messages to include data on the intentions of other vehicles. As individual vehicles can have significant effects on the total waiting time at EV charging stations, knowing the intentions of other vehicles will allow drivers to better avoid congested charging stations and achieve a lower total journey time. The performance of the system is evaluated using state-of-the-art traffic and communications simulators showing that using V2X can reduce journey times by 70%. Aside from demonstrating the performance using simulation the system has also been implemented on a real life test vehicle using functional V2X hardware to show that the system is viable for implementation.
Saxophone is a musical instrument that tends to be out of tune. I have invented an automatic tuning saxophone mouthpiece that can help saxophonists play in tune by correcting their notes' frequencies in real time during performances using a stepper motor.
...
Saxophone is a musical instrument that tends to be out of tune. I have invented an automatic tuning saxophone mouthpiece that can help saxophonists play in tune by correcting their notes' frequencies in real time during performances using a stepper motor.