This research considers using center of mass (CoM) height variation as an input for balance control on a humanoid robot. Traditional balance strategies for humanoid robots are taking a step, control of the center of pressure (CoP) location, a result of the ‘ankle strategy’, and changing the angular momentum about the CoM, for example by a ‘hip strategy’. For humanoid robots, a common assumption behind these strategies is that the CoM height is predefined. However, CoM height changes can be used as an input for balance control, as for example can be observed during the landing of an athlete after a long jump. The first contributions of this work are bounds on the initial states for the variable height inverted pendulum (VHIP) from which convergence is possible to a stopped state, also known as capture regions. First, only a unilateral contact constraint is considered; negative CoM acceleration cannot be smaller than gravitational acceleration. Second, CoM height constraints are added to the model, after which a capture region can still be computed in closed-form. Third, vertical force constraints are added, after which capture regions are computed numerically using a bang-bang control law. The last capture region bridges the transition to the applied part of this work. The second contribution is a control law on vertical acceleration, suitable for application on a humanoid robot using a momentum-based control framework. Push recovery is tested on NASA’s Valkyrie humanoid robot while the robot is standing. In simulation, an increase in recoverable push of 9% can be observed when comparing to a controller that only uses CoP, when pushing the back of the robot. On hardware, an average increase of 7% can be observed for this push direction using a load sensor. Additionally, tests are conducted on hardware on Boston Dynamics’ Atlas using a medicine ball on a rope, but no improvement in recovery is observed. The control method for standing push recovery is also extended for use while the robot is walking. For Valkyrie in simulation, recovery improved the most compared with a predefined height approach for a push applied in the first part of the single support state for rear and frontal push directions. Additionally, a hardware result on Atlas while walking is briefly presented.

Bipedal running gaits may be self-stable if they rely on the intrinsic system dynamics to attenuate deviations. This use of passive dynamics for gait stability has been observed in biological running and applied for running robotics, supported with the results of numerical simulations. It is an important aspect of the development of fast and agile legged robotic locomotion over complex terrain. One part of running stability, trunk stabilization, can be described using a virtual pendulum model. We believe such a metacentric model may be applied in robotic design to achieve passive stabilization of body pitch for fast bipedal running.

In this study, we test metacentric models for self-stabilization of body pitch in bipedal running and evaluate the effects of running speed, design parameter variations, and control strategies. We extract data from experiments with the Planar Elliptical Runner and compare these with a planar spring-loaded inverted pendulum model with a trunk (TSLIP).

We find passive self-stable gaits in the TSLIP model with the centre of mass below the hip. These gaits demonstrate robustness to disturbances and do not require inertial measurements or high-gain feedback control. At high velocities, foot placement and choice of leg stiffness become less critical for gait stability.

Data from experiments with the Planar Elliptical Runner support the findings from the model behaviour as it runs reliably, with passively stable body pitch, and without feedback control. A metacentric model, such as the virtual pendulum model, may explain the observed passive body-pitch stabilization. Evidence from the model ground reaction forces suggests that pitch stability for the Planar Elliptical Runner can also be explained with a metacentric model. Stability in height and velocity can be explained with the compliant leg behaviour in stance.

Constructing a metacentre through mechanical design is beneficial to intrinsic body-pitch stability and subsequently facilitates full gait self-stability in fast running robotics. Implicit feedback allows off-loading of high-gain feedback control onto the system mechanical design, thus contributing to developments for fast legged locomotion over rough terrain.

An important problem in reinforcement learning is the exploration-exploitation dilemma. Especially for environments with sparse or misleading rewards it has proven difficult to construct a good exploration strategy. For discrete domains good exploration strategies have been devised, but are often nontrivial to implement on more complex domains with continuous states and/or actions.
In this work, a novel persistent and directed exploration framework is developed, called Smart Start. Usually, a reinforcement learning agent executes its learned policy with some exploration strategy from the start until the end of an episode, which we call ``normal'' learning. The idea of Smart Start is to split a reinforcement learning episode in two parts, the Smart Start phase and the ``normal'' learning phase. The initial Smart Start phase guides the agent to a region in which the agent expects to learn the most. The region is constructed using previous experiences and the guiding is done using a model-based planning or trajectory optimization method. When the agent arrives at the region, it continues its ``normal'' reinforcement learning. This approach leaves the performance of the used reinforcement learning algorithm unchanged, but augments it with persistent and directed exploration.
The Smart Start framework was evaluated using three reinforcement learning algorithms, a simple model-based reinforcement learning algorithm (MBRL), R-max and Q-Learning with epsilon-greedy, Boltzmann and UCB1 exploration. The evaluation was done on four discrete gridworld environments. Three environments with sparse rewards and one with misleading rewards. We showed that the performance of Q-Learning with Smart Start is comparable to R-max, which performs near optimal in the used scenarios. The MBRL algorithm with Smart Start is even able to outperform R-max in some of the problems. We show that Smart Start is a good framework for exploration that can be incorporated with any reinforcement learning algorithm.