Quadrupedal robots possess the ability to move freely in the world and perform a variety of actions that would be unsafe or impractical for humans to perform. In the SNOW project, a quadrupedal robot is tasked with aiding firefighters in rescue missions during house fires by locating humans and assessing their health. Pushing away obstacles that cannot be circumvented otherwise is one of the many capabilities a quadrupedal robot should possess to be of most use in such missions. We develop a proof of concept by solving two problems sequentially: which stance to take on prior to the push and how to perform the push. The process of stance selection starts with generating a certain amount of stances. Stances are generated starting from a preselected stance appropriate to the goal location and deviating from the 12 joint angles with a normal distribution. All generated stances are ran through a number of filters, which rely on solving for the forward and inverse kinematics of the robot. These filters check if the initial position is sensible and balanced and if the projected final position is close to the goal and balanced. The inverse kinematics are solved using Adaptive-Network-Based Fuzzy Inference Systems (ANFIS), which results in accurate estimations within a time frame that can be used in real-time applications. The final stance is selected by comparing the total displacement of all joint angles per stance, where the lowest total displacement is considered optimal. The push is controlled by a nonlinear model predictive controller. We strive for a stable push, where the contact between the pusher and the object sticks, by keeping the movement of the end-effector within the motion cone. The motion cone denotes all twists the object can have without slipping at the contact with the pusher and is constructed using the limit surface to model the interaction between the object and the support surface and the generalized friction cone to model the interaction between the pusher and the object. We find that the motion cone predicts stick and slip with an accuracy slightly higher than 80%. Our controller steers accurately to all goals that lie within the motion cone and moves the object with a twist on the edge of the motion cone if the goal location lies outside of the motion cone. The robot remains balanced throughout the pushing motion in the vast majority of cases, but is more at risk of tipping over when pushing heavier objects.

Active knee prostheses are potent in assisting users, providing symmetry in walking, reducing metabolic costs, and preventing long-term health problems. The heart of their complex control algorithm employs the Impedance Control (IC) Law, which controls the torque output of the device by three parameters: stiffness coefficient, equilibrium angle, and damping coefficient. Ideally, these parameters should be personalized and adaptive to address interpersonal and intrapersonal variations on level-ground walking. However, current practices achieve personalization only through basic normalization and do not address adaptiveness. This thesis aims to utilize an RL framework with a novel frequency-domain state representation to address personalization and adaptiveness simultaneously. A policy iteration algorithm from the Q-learning family was chosen as the essence of the RL framework, and bellman error (BE) was chosen as the primary evaluation metric. The study revolves around two hypotheses. First, does the RL framework is suitable for the system? Second, does the frequency-domain state representation perform better than the time-domain state representation? Within the scope of the thesis, a custom environment was created by modifying the Humanoid-v04 environment of OpenAI Gym using MuJoCo (Multi-Joint dynamics with Contact). This environment is used to train the RL framework and conduct the experiments. Results suggest that the proposed RL framework can improve the system, and the frequency-domain state representation is superior to its time-domain counterpart. The latter conclusion has an impact beyond the active knee prosthesis domain and can inspire any trajectory following tasks with periodic signals.

Greenhouses allow production of crops that would otherwise be impossible. Permitting more local, fresher and nutrient richer crop production. Eorts are taken to minimize societal harm due to energy and resource consumption by greenhouse production systems. One way to control such systems is by using model predictive control. Optimal crop yield and resource eciency can, in theory, be achieved by model predictive control. Unfortunately, one major drawback of model predictive control is that it is not well equipped to deal with parametric uncertainty. Significant prediction errors can occur when a mismatch between the model and the real system exists, resulting in deteriorated performance of the system. Strategies exist, such as robust MPC, that are designed to handle uncertainty, but those often result in conservative control policies. This thesis proposes to use model predictive control as a function approximator for RL in order to learn values for model and MPC parameters that can deliver optimal performance in the case of model mismatch. In this thesis, data-driven economic nonlinear model predictive control using Q-learning is proposed as a method to alter the model parameters. The performance of the system af- ter learning is compared to approaches using robust and nominal model predictive control. Three dierent goals are determined: maximizing economic profit, minimizing the constraint violations and maximizing the economic performance while minimizing constraint violations. In this work, an ENMPC scheme is used as a function approximator in a Q-learning envi- ronment. The optimization solution from the ENMPC scheme is used as the input to the system, while the Q-learning agent optimizes the parameter values of the ENMPC scheme and model for the environment. The performance of the system after learning is compared to approaches using robust and nominal model predictive control. The simulation results show that the data-driven ENMPC using reinforcement learning is able to decrease constraint vi- olations by up to 94%, but unable to increase economic performance compared to nominal MPC, compared to robust MPC the EPI is increased by almost 10% while keeping constraint violations at a similar level.

The renewable generation capacity, particularly solar and wind power installations, has increased steadily in the Netherlands over the course of recent years. Due to the local, small-scale nature of these power plants (compared to conventional power plants), a large share of this generation capacity is installed into the medium- and low-voltage distribution grids. This trend acts as the source of several challenges for distribution system operators (DSOs) such as Stedin Netbeheer B.V. (a DSO in the Netherlands, supporting this project). One of the main problems is the fact that voltage limit violations, particularly upper voltage limit violations, become more frequent. Conventional voltage control schemes assume a unidirectional power flow (i.e., consumption only) in distribution grids and are unable to keep the limits in case a large share of generation capacity is installed. This is further complicated by the fact that underground cable networks, typical in Western European electricity distribution grids, have lines with large R/X ratios, which reduces the effectiveness of reactive power injection-based voltage control methods. This MSc project intends to solve the issue of voltage limit violations with a model predictive control (MPC) policy. The considered control actions coordinated by the model predictive controller are the switching of the on-load tap changer (OLTC) mechanism mounted to the primary substation’s transformer, setpoint adjustments of the low-level OLTC control relay, and the active power curtailment of larger photovoltaic plants. A linear, sensitivity-matrix-based model is used for the grid’s state prediction; and the sensitivity values are re-calculated at each sampling time step of the MPC. To avoid the curtailment of photovoltaic (PV) plants when not justified, a conditional curtailment logic is incorporated into the MPC policy: PV plants are only allowed to be curtailed if their local voltage magnitude is above a tuneable threshold. This logic is described and incorporated into the model predictive controller’s optimization problem with the help of binary variables and mixed-integer linear (MIL) constraints. The benefit of incorporating knowledge about future disturbances (load, generation, and external grid voltage profiles) is also tested, in order to assess the potential benefit DSOs could get from forecasting these quantities. A case study was conducted in which the considered controllers were tested on a section of a Stedin grid that carries the characteristics of a typical Western European medium voltage distribution grid: large R/X ratios and a large installed PV generation capacity both in the form of household generation and larger PV plants. All controllers were simulated in 4 different test cases: a typical summer day, a summer day with the external grid’s overvoltage, a summer day with 2 out of 3 large PV plants not operating, and a typical winter day. All test cases have the control goals of mitigating limit violations and ensuring that the nodal voltage magnitudes are as close as possible to the nominal 1 per unit throughout the day. The winter day test case has the additional goal of avoiding excessive curtailment as PV energy is worth considerably more during these days. All profile data is based on real Stedin measurements. The designed model predictive control policies are compared to two simple control schemes: current compounding, i.e. when the primary substation’s automatic voltage control relay’s setpoint is adjusted based on the active power delivery through the substation’s transformer, and another scheme when current compounding is combined with local active power curtailment controllers for large PV plants. The most important metrics used for comparison are voltage root mean square error (RMSE), the total voltage limit violation area, the percentage of curtailed PV energy, and the number of tap changes over the considered day. The simulations were carried out using Python and DIgSILENT PowerFactory. The simulation results show that the MPC policy can perform better than the simple control schemes but only when exact knowledge of future profiles is available. In this case, the MPC results in lower voltage RMSE, smaller violation areas, and lower curtailment percentage values, at the expense of using more tap changes in all 4 test cases. In 3 of the 4 test cases, MPC was completely able to eliminate voltage limit violations, showing clearly the advantages of good quality forecasts on future disturbances. Since this exact knowledge about the near future fluctuations is quite ideal, more realistic MPC policies were also tested with no future knowledge and tightened voltage constraints. These simulations brought mixed results when compared with the simple schemes, performing better in terms of voltage RMSE, but worse in terms of the total voltage limit violation area.