In this paper, a novel active vision strategy is proposed for optimizing the viewpoint of a robot's vision sensor for a given success criterion. The strategy is based on extremum seeking control (ESC), which introduces two main advantages: 1) Our approach is model free: It does not require an explicit objective function or any other task model to calculate the gradient direction for viewpoint optimization. This brings new possibilities for the use of active vision in unstructured environments, since a priori knowledge of the surroundings and the target objects is not required. 2) ESC conducts continuous optimization backed up with mechanisms to escape from local maxima. This enables an efficient execution of an active vision task. We demonstrate our approach with two applications in the object recognition and manipulation fields, where the model-free approach brings various benefits: for object recognition, our framework removes the dependence on offline training data for viewpoint optimization, and provides robustness of the system to occlusions and changing lighting conditions. In object manipulation, the model-free approach allows us to increase the success rate of a grasp synthesis algorithm without the need of an object model; the algorithm only uses continuous measurements of the objective value, i.e., the grasp quality. Our experiments show that continuous viewpoint optimization can efficiently increase the data quality for the underlying algorithm, while maintaining the robustness.@en

Reinforcement learning techniques enable robots to deal with their own dynamics and with unknown environments without using explicit models or preprogrammed behaviors. However, reinforcement learning relies on intrinsically risky exploration, which is often damaging for physical systems. In the case of the bipedal walking robot Leo, which is studied in this paper, two sources of damage can be identified: fatigue of gearboxes due to backlash re-engagements, and the overall system damage due to falls of the robot. We investigate several exploration techniques and compare them in terms of gearbox fatigue, cumulative number of falls and undiscounted return. The results show that exploration with the Ornstein-Uhlenbeck (OU) process noise leads to the highest return, but at the same time it causes the largest number of falls. The Previous Action-Dependent Action (PADA) method results in drastically reduced fatigue, but also a large number of falls. The results reveal a previously unknown trade-off between the two sources of damage. Inspired by the OU and PADA methods, we propose four new action-selection methods in a systematic way. One of the proposed methods with a time-correlated noise outperforms the well-known e-greedy method in all three benchmarks. We provide guidance towards the choice of exploration strategy for reinforcement learning applications on real physical systems.@en

Model-free reinforcement learning and nonlinear model predictive control are two different approaches for controlling a dynamic system in an optimal way according to a prescribed cost function. Reinforcement learning acquires a control policy through exploratory interaction with the system, while nonlinear model predictive control exploits an explicitly given mathematical model of the system. In this article, we provide a comprehensive comparison of the performance of reinforcement learning and nonlinear model predictive control for an ideal system as well as for a system with parametric and structural uncertainties. The comparison is based on two different criteria, namely the similarity of trajectories and the resulting rewards. The evaluation of both methods is performed on a standard benchmark problem: a cart–pendulum swing-up and balance task. We first find suitable mathematical formulations and discuss the effect of the differences in the problem formulations. Then, we investigate the robustness of reinforcement learning and nonlinear model predictive control against uncertainties. The results demonstrate that nonlinear model predictive control has advantages over reinforcement learning if uncertainties can be eliminated through identification of the system parameters. Otherwise, there exists a break-even point after which model-free reinforcement learning performs better than nonlinear model predictive control with an inaccurate model. These findings suggest that benefits can be obtained by combining these methods for real systems being subject to such uncertainties. In the future, we plan to develop a hybrid controller and evaluate its performance on a real seven-degree-of-freedom walking robot.@en

Abstract poster: https://personalrobotics.ri.cmu.edu/RSS2013Uncertainty/abstract5.pdf@en

The advent and subsequent popularity of low cost, low power CMOS vision sensors enables us to integrate processing logic on the camera chip itself, thereby creating so-called smart sensors. They have an on-chip SIMD data processing array controlled by an off-chip controller. Smart sensors can execute low-level image processing routines as soon as one or more image lines are converted; they do not have to wait for the whole image. High level image processing like feature extraction and object detection and tracking can be performed with a separate powerful off-chip processor. Current solutions have several problems. It is totally unclear what the right architectural parameters are for a given application domain. There are many parameters, like pixel array size, pixel properties, number of AD-converter units, accuracy, number of pixels per SIMD processor element, processor element functionality, etc. Also the functionality of the external processor and its connectivity with the smart sensor has to be determined. Furthermore, intuitive mappings of algorithms on architecture components are used, after the architecture has been determined. It will be clear that this is far from optimal. Finally we foresee a further integration, making a combination of on-chip vision sensor, pixel processing, control, and feature / object processing possible. The result is a low-cost one-chip smart camera (so-called SmartCam) solution. This means that research is needed to explore the new architectural opportunities and consequences. The SmartCam project investigates these new opportunities and contributes to a better and more quantitatively guided design trajectory. In particular, we will investigate the impact of current applications, de¯ne relevant architectural parameters and develop an architectural template, enhance our existing application mapping environments for SIMD and ILP (Instruction-Level Parallel) processors, and perform two case studies. The work will focus on creating an environment for exploring the design space parametrized by the architectural template and integrating this with our application mapping environment. Keywords: Embedded system design, Smart Sensors, Design Space Exploration, Parametric design, Programmable design, Heterogeneous architecture, Low power, High performance, Small and Cheap implementations@en

Learning-based approaches are suitable for the control of systems with unknown dynamics. However, learning from scratch involves many trials with exploratory actions until a good control policy is discovered. Real robots usually cannot withstand the exploratory actions and suffer damage. This problem can be circumvented by combining learning with a model-based control. In this letter, we employ a nominal model-predictive controller that is impeded by the presence of an unknown model-plant mismatch. To compensate for the mismatch, we propose two approaches of combining reinforcement learning with the nominal controller. The first approach learns a compensatory control action that minimizes the same performance measure as is minimized by the nominal controller. The second approach learns a compensatory signal from a difference of a transition predicted by the internal model and an actual transition. We compare the approaches on a robot attached to the ground and performing a setpoint reaching task in simulations. We implement the better approach on the real robot and demonstrate successful learning results.@en