Deep reinforcement learning (DRL) and evolution strategies (ESs) have surpassed human-level control in many sequential decision-making problems, yet many open challenges still exist. To get insights into the strengths and weaknesses of DRL versus ESs, an analysis of their respective capabilities and limitations is provided. After presenting their fundamental concepts and algorithms, a comparison is provided on key aspects, such as scalability, exploration, adaptation to dynamic environments, and multiagent learning. Current research challenges are also discussed, including sample efficiency, exploration versus exploitation, dealing with sparse rewards, and learning to plan. Then, the benefits of hybrid algorithms that combine DRL and ESs are highlighted.

Energy-neutral Internet of Things requires freeing embedded devices from batteries and powering them from ambient energy. Ambient energy is, however, unpredictable and can only power a device intermittently. Therefore, the paradigm of intermittent execution is to save the program state into non-volatile memory frequently to preserve the execution progress. In task-based intermittent programming, the state is saved at task transition. Tasks are fixed at compile time and agnostic to energy conditions. Thus, the state may be saved either more often than necessary or not often enough for the program to progress and terminate. To address these challenges, we propose Coala, an adaptive and efficient task-based execution model. Coala progresses on a multi-task scale when energy permits and preserves the computation progress on a sub-task scale if necessary. Coala's specialized memory virtualization mechanism ensures that power failures do not leave the program state in non-volatile memory inconsistent. Our evaluation on a real energy-harvesting platform not only shows that Coala reduces runtime by up to 54% as compared to a state-of-the-art system, but also it is able to progress where static systems fail.

The main obstacles to achieve truly ubiquitous sensing are (i) the limitations of battery technology - batteries are short-lived, hazardous, bulky, and costly - and (ii) the unpredictability of ambient power. The latter causes sensors to operate intermittently, violating the availability requirements of many real-world applications. In this paper, we present the Coalesced Intermittent Sensor (CIS), an intermittently-powered sensor that senses continuously! Although a single node will frequently be off charging, a group of nodes can -in principle- sense 24/7 provided that their awake times are spread apart. As communication is too expensive, we rely on inherent component variations that induce small differences in power cycles. This basic assumption has been verified through measurements of different nodes and power sources. However, desynchronizing nodes is not enough. An important finding is that a CIS designed for certain (minimal) energy conditions will become synchronized when the available energy exceeds the design point. Nodes employing a sleep mode (to extend their availability) do wake up collectively at some event, process it, and return to charging as the remaining energy is typically too low to handle another event. This results in multiple responses (bad) and missing subsequent events (worse) due to the synchronized charging. To counter this undesired behavior we designed an algorithm to estimate the number of active neighbors and respond proportionally to an event. We show that when intermittent nodes randomize their responses to events, in favorable energy conditions, the CIS reduces the duplicated captured events by 50% and increases the percentage of capturing entire bursts above 85%.

—Battery-free computational RFID platforms, such as WISP (Wireless Identification and Sensing Platform), are intermittently-powered devices designed for replacing existing sensor networks. Accordingly, synchronization appears as one of the crucial building blocks for collaborative and coordinated actions in these platforms. However, intermittent power leads to frequent loss of computational state and short-term clock frequency instability that makes synchronization challenging. In this article, we introduce the WISP-Sync protocol that provides synchronization among WISP tags in the communication range of an RFID reader. WISP-Sync overcomes the aforementioned challenges by employing a Proportional-Integral (PI) controller-inspired algorithm which (i) is adaptive—reactive to short-term clock instabilities; (ii) requires only a few computation steps—suitable for limited harvested energy; and (iii) keeps a few variables to hold the synchronization state—minimum overhead to recover from power interrupts. Evaluations in our testbed showed that WISP-Sync ensured an average synchronization error of approximately 1 ms among the tags with an average energy overhead of 1.85 mJ per synchronization round.

Tiny energy harvesting battery-free devices promise maintenance free operation for decades, providing swarm scale intelligence in applications from healthcare to building monitoring. These devices operate intermittently because of unpredictable, dynamic energy harvesting environments, failing when energy is scarce. Despite this dynamic operation, current programming models are static; they ignore the event-driven and time-sensitive nature of sensing applications, focusing only on preserving forward progress while maintaining performance. This paper proposes InK; the first reactive kernel that provides a novel way to program these tiny energy harvesting devices that focuses on their main application of event-driven sensing. InK brings an event-driven paradigm shift for batteryless applications, introducing building blocks and abstractions that enable reacting to changes in available energy and variations in sensing data, alongside task scheduling, while maintaining a consistent memory and sense of time. We implemented several event-driven applications for InK, conducted a user study, and benchmarked InK against the state-of-the-art; InK provides up to 14 times more responsiveness and was easier to use. We show that InK enables never before seen batteryless applications, and facilitates more sophisticated batteryless programs.

Backscatter has emerged as the dominant paradigm for battery-free networking among the (potentially) trillions of devices in the future Internet of Things, partly because of the order of magnitude smaller energy consumption, but at the cost of collisions, low data rates, and short distances. This position paper explores the alternative approach: using lowpower, yet active radios to communicate among the battery-less swarm. We describe the challenges of using active radios in this context, including lack of tight time guarantees, high listening costs, and intermittent operation. While backscatter is promising, this paper hopes to broaden the conversation around alternative methods for networking the future IoT.

Energy harvesting and battery-free sensing devices show great promise for revolutionizing computing in the home, in the wild, and on the body. The promise of cheap, dense, and ubiquitous sensing technology brings new applications for the Internet of Things. However, the future programming model is blurry and complex. With a potential for trillions of devices, and thousands of devices per person on earth, programming languages and associated operating systems must be usable, flexible, and resource efficient. Because of the thousands of applications and fine grained differences in requirements, multi-tenancy may be a part of the solution to solving this programming model crisis. This paper explores the energy and resources costs, feasibility, and motivation for multi-tenancy on these tiniest of computing devices-namely the difficulties in scheduling tasks fairly, efficiently, and simply. Because of intermittent power, resources and energy must be mostly devoted towards user tasks, we implement a rudimentary operating system with low overhead to conduct experiments and test time-sharing and scheduling protocols. We close with a discussion on challenges to implementing a multi-tenant run-time on battery-free tags, and proposals for future work.