Haptic teleoperation is a promising technology with applications in telemaintenance and disaster management. However, it faces significant challenges when the application is subjected to a high network latency and environments with moving objects. This work aims to extend Model Mediated Teleoperation (MMT) to overcome challenges in supporting dynamic environments. Instead of striving for perfect model alignment, we acknowledge the inevitable mismatch between the remote environment and its model at the operator. We propose a set of design principles and an accompanying framework for designing MMT solutions that prioritize operator intent. Our approach is exemplified through an application where an operator, located 8000 km away (The Netherlands - India) and subjected to an average of 179 ms end-to-end latency, guides a robot arm to draw on a whiteboard whose position is actively altered. We evaluate the effectiveness of our approach through a user study. We show a 3-point improvement on a 7-point Likert scale when users utilize our approach to teleoperate over significant network latency of up to 1 s.

After the potential of this work is realized, people will be able to physically manipulate remote environments. For example, a skilled artist in Tokyo could paint delicate calligraphy on a canvas in Paris, feeling each stroke as if they were local. A surgeon in London could operate on a patient in a remote village, sensing the precise resistance of tissue through robotic instruments. A firefighter in Los Angeles could save people from a burning building without the need to put his own life at stake. Extending our human touch across great distances opens doors to new forms of work, collaboration, and human connection without needing physical presence.

Realizing this vision requires the successful implementation of Haptic Bilateral Teleoperation (HBT). An HBT system must fulfill two core requirements: precise replication of the operator’s actions by a remote robot and accurate, responsive feedback to guide those actions. These requirements are inherently subjective, varying across individuals, tasks, and applications, adding significant complexity to both the system design and evaluation. At first glance, realizing HBT may seem an insurmountable challenge. Conventional wisdom suggests that the stringent network requirements, such as ultra-low latency and near-perfect reliability, far exceed the capabilities of current network technology. The latency constraints are so strict that even fundamental physical limits, such as the speed of light, impose onerous restrictions on the maximum feasible distance between the operator and the remote environment.

Overcoming these challenges demands a holistic approach. On the one hand, we must push network technology to its limits, striving for lower latency, higher reliability, and optimized communication protocols explicitly tailored for HBT applications. On the other hand, we must also explore alternative approaches that lower the network requirements of HBT systems, especially the latency requirement. For both of these directions, it is essential to have a deep understanding of the entire HBT system, particularly the role of the human operator. Unlike most systems, where performance is measured through objective metrics, HBT introduces a distinctive challenge: HBT systems must be designed for both technical performance and the user’s subjective experience.

In this dissertation, we first provide a deeper understanding of HBT systems and examine how network behavior influences user experience. In particular, we identify the underlying reasons behind the stringent network requirements. First, through multiple repeated user studies, we demonstrate that the reliability of the kinematic demands and force modalities is low, especially at the packet rate 1 kHz. Even with 50%, packet loss, we demonstrate that users are largely unaffected due to strong temporal correlation in these modalities.

More importantly, we pinpoint the fundamental cause of the strict low-latency requirement. It is not merely the presence of delay but rather the unintended forces that arise due to the combination of active force feedback and a closed-loop control system. This interaction is unique because users do not perceive latency directly. Instead, they experience the resulting unnatural forces.

Because the main cause for the stringent network requirements is so specific, it provides a clear target for research. Next, we explore multiple approaches to address this particular interaction, which is the primary source of stringent latency constraints. First, we optimize the MAC protocols with a strict focus on minimizing latency for both the kinematic and force modalities. Next, we investigate methods to manipulate the transmitted data in a way that does not impede the human operator, aiming to mitigate the adverse effects of network latency on force feedback. Finally, we take a more radical approach by moving away from direct transmission of force feedback altogether, instead leveraging predictive models to estimate force feedback locally.

An important insight from this dissertation is the path forward for HBT systems. Future HBT systems should integrate predictive force feedback with live video transmission, leveraging the advantages of each modality. Predictive force feedback offers a viable alternative to the stringent latency constraints of transmitted force feedback. Minor inaccuracies in force feedback are often imperceptible to human operators. Meanwhile, live video transmission circumvents the complexities of visual prediction while operating within a latency range of approximately 100ms. This is significantly more feasible than the 1ms latency required for direct force feedback transmissions.

This dissertation has three important takeaways. First, it provides a deeper understanding of how network performance shapes user experience in HBT. Second, it demonstrates alternative approaches that enable HBT beyond direct network improvements. Third, it proposes a path forward that integrates live video with predictive force feedback. Despite these advancements, significant challenges remain. Scaling HBT to highly dynamic environments, where unpredictability complicates prediction of force feedback, remains a major hurdle. Additionally, managing discrepancies between the operator’s predictive experience and the actual remote events is crucial to maintaining intuitive and stable interactions. While these challenges persist, none appear insurmountable. With continued progress, HBT can become a transformative technology, opening doors to new forms of work, collaboration, and human connection without needing physical presence.

Haptic bilateral teleoperation holds promise for applications such as telemaintenance, remote manipulation, and disaster response, yet delivering precise, low-latency force and video feedback remains challenging. This study advances haptic bilateral teleoperation by combining live video with Model Mediated Teleoperation (MMT) to enable predictive force feedback. While this method has benefits, several non-trivial challenges, such as synchronizing the model with user's and remote robot's actions, arise. A novel algorithm is developed that allows the robotic device to replicate interactions predictively experienced by the operator. We validated this approach in a fully functional system that performs reliably despite significant network delays. The latency performance of the system is extensively characterized, achieving a motion-to-pixel latency of 58 ms. A user study revealed that operators did not perceive network latency of at least 75 ms, resulting in a 133 ms motion-to-pixel delay requirement. Additionally, a 5G latency analysis demonstrated that effective haptic teleoperation is achievable with both operator and remote ends connected via 5G. This provides a path away from strict latency requirements toward practical teleoperation solutions using currently available technology.

The next frontier in communications is teleoperation - manipulation and control of remote environments with haptic feedback. Compared to conventional networked applications, teleoperation poses widely different requirements, ultra-low latency (ULL) is primary. Realizing ULL communication demands significant redesign of conventional networking techniques, and the network infrastructure envisioned for achieving this is termed as Tactile Internet (TI). The design of meaningful performance metrics is crucial for seamless TI communication. However, existing performance metrics fall severely short of comprehensively characterizing TI performance due to their inability to capture how well sensed signals are reproduced. We take Dynamic Time Warping(DTW) as the basis of our work and identify necessary changes for characterizing TI performance. Through substantial refinements to DTW, we design Effective Time- and Value-Offset (ETVO) - a new method for measuring the fine-grained performance of TI systems. Through an in-depth objective analysis, we demonstrate the improvements of ETVO over DTW. Through subjective experiments, we demonstrate that existing QoS and QoE methods fall short of estimating the TI session performance accurately. Using subjective experiments, we demonstrate the behavior of the proposed metrics, their ability to match theoretically derived performance, and finally, their ability to reflect user satisfaction in a practical setting.

Tactile Internet (TI) envisions communicating haptic sensory information and kinesthetic feedback over the network and is expected to transfer human skills remotely. For mission-critical TI applications, the network latency is commonly mandated to be between 1-10 ms, due to the sensitivity of human touch, and the packet delivery ratio to be 99.99999%, failing which can lead to catastrophic outcomes. However, with humans-in-the-loop, their dexterity and adaptability to varying responses to stimuli under different network conditions, measuring the performance of a TI session only with latency and packet losses are insufficient and presents an incorrect representation of the experience of the TI application. To develop an objective measure of the quality of TI sessions, we propose a framework that models TI applications as networked control systems, including humans-in-the-loop. We derive a closed-form expression for measuring the difference between the application performance in ideal and non-ideal network conditions. Based on Weber’s law of Just Noticeable Difference, we provide a metric called TIM to estimate the impact of the network on haptic feedback. We implemented TIM on multiple applications on a TI testbed to show that our approach is feasible and TIM strongly follows real subjective measurements. Further, we propose a channel compensation spring based on TIM, to alleviate the network conditions’ negative effects. We demonstrate the efficacy of the channel compensation spring in improving the user experience. We also present implementation notes for TI application developers.

The pioneering field of tactile Internet (TI) will enable the transfer of human skills over long distances through haptic feedback. Realizing this demands a roundtrip latency of sub-5 ms. In this work, we investigate the capability of Wi-Fi 6 and existing TI scheduling/multiplexing schemes in meeting this stringent latency constraint. Taking the concrete example of the state-of-the-art video-haptic multiplexer (VH-multiplexer), we highlight the pitfalls of relying on the existing Wi-Fi 6 systems for TI communication. To circumvent this, we propose video-tactile latency scheduler (ViTaLS) - a novel link layer framework for tuning the video-tactile frame transmissions to suit their heterogeneous Quality of Service requirements. We present a mathematical model to characterize the packet transmission duration of ViTaLS. Using a custom simulator, we validate our model and measure the objective performance improvement of ViTaLS over VH-multiplexer. We also present ViTaLS-optimal - a variant of ViTaLS, for further 4 reducing the tactile latency. Objectively, we show that ViTaLS-optimal yields a latency improvement of up to 82 %. Based on experiments conducted on a real TI testbed, we subjectively demonstrate that ViTaLS-optimal outperforms the VH-multiplexer.

The next frontier for immersive applications is enabling sentience over the Internet. Tactile Internet (TI) envisages transporting skills by providing ultra-low-latency (ULL) communications for transporting touch senses. In this work, we focus our study on the first/last mile communication, where the future generation WiFi-7 is pitched as the front-runner for ULL applications. We discuss a few candidate features of WiFi-7 and highlight its major pitfalls with respect to ULL communication. Further, through a specific implementation of WiFi-7 (vanilla WiFi-7) in our custom simulator, we demonstrate the impact of one of the pitfalls - the standard practice of using jitter buffer in conjunction with frame aggregation - on TI communication. To circumvent this, we propose the Non-Buffered Scheme (NoBuS) - a simple MAC layer enhancement for enabling TI applications over WiFi-7. NoBuS trades off packet loss for latency, enabling swift synchronization between the master and controlled domains. Our findings reveal that employing NoBuS yields a significant improvement in RMSE of TI signals. Further, we show that the worst case WiFi latency with NoBuS is 3.72 ms - an order of magnitude lower than vanilla WiFi-7 even under highly congested network conditions.

The next frontier in communications is teleoperation - manipulation and control of remote environments. Compared to conventional networked applications, teleoperation poses widely different requirements, ultra-low latency (ULL) being the primary one. Teleoperation, along with a host of other applications requiring ULL communication, is termed as Tactile Internet (TI). A significant redesign of conventional networking techniques is necessary to realize TI applications. Further, these advancements can be evaluated only when meaningful performance metrics are available. However, existing TI performance metrics fall severely short of comprehensively characterizing TI performance. In this paper, we take the first step towards bridging this gap. To this end, we propose a method that captures the fine-grained performance of TI in terms of delay and precision. We take Dynamic Time Warping (DTW) as the basis of our work and identify whether it is sufficient in characterizing TI systems. We refine DTW by developing a framework called Effective Time- and Value-Offset (ETVO) that extracts fine-grained time and value offsets between input and output signals of TI. Using ETVO, we present two quantitative metrics for TI - Effective Delay-Derivative (EDD) and Effective Root Mean Square Error. Through rigorous experiments conducted on a realistic TI setup, we demonstrate the potential of the proposed metrics to precisely characterize TI interactions.

In recent years, enormous growth has been witnessed in the computational and storage capabilities of mobile devices. However, much of this computational and storage capabilities are not always fully used. On the other hand, popularity of mobile edge computing which aims to replace the traditional centralized powerful cloud with multiple edge servers is rapidly growing. In particular, applications having strict latency requirements can be best served by the mobile edge clouds due to a reduced round-trip delay. In this paper we propose a Multi-Path TCP (MPTCP) enabled mobile device cloud (MDC) as a replacement to the existing TCP based or D2D device cloud techniques, as it effectively makes use of the available bandwidth by providing much higher throughput as well as ensures robust wireless connectivity. We investigate the congestion in mobile-device cloud formation resulting mainly due to the message passing for service providing nodes at the time of discovery, service continuity and formation of cloud composition. We propose a user space agent called congestion handler that enable offloading of packets from one sub-flow to the other under link quality constraints. Further, we discuss the benefits of this design and perform preliminary analysis of the system.