· · · Institutional Repository

It is suspected that instability problems in the current generation of lung simulators are caused by its actuator, a brushless DC motor, in combination with the system configuration. The hypothesis is that these problems can be resolved by replacing the actuator with a backdrivable actuator (that is, an actuator that responds well to external force) in a new system. In this BSc Thesis this hypothesis is researched. The backdrivable actuator (in this particular case, a Voice Coil actuator) in a new system can overcome the instability problems.

De professionele paraglider en fotograaf Leo Westerkamp is instaat de kijker dichter bij de natuur en sport te brengen dan ieder ander log en niet wendbaar apparaat. Echter door bochten, windvlagen, luchtdrukverschillen en trillingen van de motor ondervinden de beelden een ernstig stabiliteitsprobleem. Vandaar dan ook dat Featherweight de opdracht heeft gekregen om de camerabeelden zodanig te stabiliseren dat de beelden te verkopen zijn. Tijdens het ontwerpproces is er hoofdzakelijk rekening gehouden met de mogelijkheid de camerastabilisator lichtgewicht en makkelijk bestuurbaar te maken. Aan de andere kant mocht hierbij niet worden ingeleverd op de kwaliteit van de beelden. Ontwerp Om te voldoen aan de wensen van Westerkamp heeft Featherweight gekozen voor de Featherweight Professional. De Featherweight Professional bestaat uit een opzet waar de camera gestabiliseerd wordt door twee gekoppelde dc motoren. Westerkamp heeft zelf de controle om de camera ergens anders op te richten via een duimcontroller, en via de aangeraden LCD-bril is hij instaat zelf real-time te kijken waar hij de camera op heeft gefocust. De motoren worden aangestuurd door een laptop, wiens data verkregen wordt door een microcontroller (Arduino-Mega) en de positie van de duimcontroller. De Arduino-Mega is er om de data van de acceleratiemeter en gyrometer om te zetten naar de afwijkende hoeken die ontstaan zijn door stabilisatie problemen. Implementatie De data van de acceleratiemeter en gyrometer wordt via een I2C protocol naar de Arduino-Mega gestuurd. Hier wordt door gebruik te maken van een versimpelde Kalman filter de verkregen data omgezet naar het aantal graden dat de camera afwijkt van het punt waar hij op gestabiliseerd moet worden. De Arduino-Mega stuurt het op zijn beurt door naar de laptop met de RS-232 interface, waar het wordt verwerkt met MATLAB. Bij MATLAB ligt de nadruk op: - De hoeken aanpassen door de data van de duimcontroller en de Arduino Mega te combineren; - De hoeken begrenzen; - De hoeken en hoeksnelheden via een adaptieve PD regelaar om te zetten naar het gewenste koppel, waar de twee RX-64 motoren mee aangestuurd worden. Daarnaast worden de huidige beelden waar de camera opgericht staat, weergegeven via een LCD-scherm. Evaluatie Bij de evaluatie kwam naar voren dat veel van de eisen uit het programma van eisen voldaan werden, voor zover deze geïmplementeerd waren in het prototype. Het systeem is licht en compact genoeg en stabiliseert de grootste verstoringen uit het beeld. Daarnaast is het systeem makkelijk te bedienen en niet hinderlijk voor de piloot. Aanbevelingen Er wordt aanbevolen, voordat de Featherweight Professional in ontwikkeling wordt genomen, nader onderzoek te doen naar de gebruikerswensen, toepassen van een normale Kalman filter op de sensordata, en het omlaag brengen van de verwerkingstijd. Ook wordt er aangeraden om mogelijk over te stappen naar een gimbal constructie, het toepassen van een lock-functie zodat de camera altijd het gewenste object volgt, en het mogelijk toepassen van een softwarematige stabilisatie. Om Featherweight Professional op een bredere markt te verkopen, moet er onderzoek gedaan worden naar mogelijke klanten. Dit onderzoek kan gedaan worden door een gebruikersonderzoek uit te voeren op een testgroep.

A Trailing Suction Hopper Dredger (TSHD) is a ship that excavates sediments from the sea bottom while sailing. In situ material is excavated with a special tool called the Drag-Head, then it is hydraulically transported through a pipe to the hopper where it is temporarily stored. After the dredging is completed the collected material is transported and discharged at a specified location. The efficiency of this process is highly dependent on the detailed knowledge of the excavated soil. The optimization of dredging operations is of vital importance for future improvement in efficiency, accuracy and from the viewpoint of labor saving. The automated onboard systems that have been developed to optimize the dredging performance require knowledge of several uncertain soil-dependent parameters. These cannot be directly measured but have to be estimated online from the available measurements. Such estimation is a challenging task due to lack of sufficient sensors, severe nonlinearities in models, and time-varying nature of the parameters of interest. In this thesis we focus on two of the most important TSHD-related models. These are: I. Drag-Head Model - describing the excavation process, II. Hopper Model - describing the sedimentation process occurring inside the hopper. They contain several uncertain soil-dependent parameters that need to be estimated. These are: I. horizontal cutting force coefficient kch (Drag-Head Model ), II. ratio kvh between the horizontal and vertical cutting forces (Drag-Head Model ), III. in situ permeability ksi (Drag-Head Model ), IV. average grain diameter dm (Hopper Model ). Both processes, together with the corresponding estimation problems, are discussed in detail in Chapter 2. The highly uncertain and time-varying nature of the soil-dependent parameters and the nonlinear dynamics of the models used to describe dredging process make the estimation a challenging task. The algorithms that are capable of tackling these type of problems are Nonlinear Bayesian Filters (NBF). In Chapter 3 we review several types of NBF, namely: I. parametric filters based on the Taylor series expansion (EKF, IEKF), II. parametric filters based on statistical approximations (UKF, GHF, CDF), III. parametric filters based on Gaussian Sum approximations (GSF), IV. nonparametric filters based on the importance sampling (BPF), V. nonparametric filters based on the mean-field control-oriented approach (FPF). In Chapter 4 we investigate the applicability of these nonlinear filters to the estimation problems that originate from the Drag-Head Model. The problems are: the Cutting Estimation Problem and the Cutting and Jetting Estimation Problem. The Cutting Estimation Problem applies for any cutting excavation tool whereas the Cutting and Jetting Estimation Problem is applicable only for tools equipped with cutting and jetting components. The former problem considers estimation of the ratio kvh between cutting forces and the horizontal cutting force coefficient kch, the latter problem deals with the estimation of the horizontal cutting force coefficient kch and the in situ permeability ksi. To solve the aforementioned estimation problems one needs to handle time-varying delay in the measurement of incoming density ρi, which is discussed separately. It is concluded that among the tested methods the best solution to the Cutting Estimation Problem is provided by the CDF and, in case of large uncertainty in the initial states, by the GSF. To solve the Cutting and Jetting Estimation Problem it is crucial to exploit the correlation between the horizontal cutting force coefficient kch and the in situ permeability ksi. This is done by a cascaded filter, which uses the PF to obtain an estimate of ksi, which will be further filtered by a Steady State Identification (SSI) filter, and finally by the BF to produce a final estimate of kch. In Chapter 5 we develop a novel class of nonlinear particle filters: the Saturated Particle Filter (SPF) that is used to solve the Hopper Estimation Problem. The SPF is a general method designed for Saturated Stochastic Dynamical Systems (SSDS), which are severely nonlinear systems often used in modeling real-life problems. They are characterized by a constrained probability distribution exhibiting singularity on the boundary of the saturation region. Such singularities make it difficult to estimate the states or the parameters of SSDSs by standard nonlinear filters. Our new method exploits the specific structure of the SSDS in order to design an importance sampling distribution that accounts for the most recent measurements in the prediction step of the filtering algorithm. Chapter 6 deals with the asymptotic properties of the SPF. We establish the conditions under which the SPF converges to the optimal theoretical filter. The convergence of our method is closely related to the appropriate resampling scheme. This led to the development of the improved Saturated Particle Filter (iSPF) which combines the importance sampling of the SPF with a novel resampling algorithm. In Chapter 7 the iSPF together with other nonparametric methods from Chapter 3 are used to estimate the average grain diameter dm, which solves the Hopper Estimation Problem. Because the sedimentation process is naturally divided into three regimes, to find the most efficient filtering method we considered each mode separately. We conclude that: I. for the No-Overflow loading phase the best estimate of dm is obtained by the FPF, II. for the Overflow loading phases with weak erosion, the recommended filtering method is the Reduced-Order PF, III. for the Overflow loading phases with strong erosion, the best estimation performance is achieved by the Reduced-Order PF when the excavated soil is fine and the Hybrid SPF when the excavated soil is coarse. The final solution to the Hopper Estimation Problem is obtained by integrating the filters designed for separate modes into a global estimator. Chapter 8 concludes the thesis.

Reinforcement Learning (RL) is a popular method in machine learning. In RL, an agent learns a policy by observing state-transitions and receiving feedback in the form of a reward signal. The learning problem can be solved by interaction with the system only, without prior knowledge of that system. However, real-time learning from interaction with the system only, leads to slow learning as every time-interval can only be used to observe a single state-transition. Learning can by accelerated by using a Dyna-style algorithm. This approach learns from interaction with the real system and a model of that system simultaneously. Our research investigates two aspects of this method: Building a model during learning and implementing this model into the learning algorithm. We use a memory-based modeling method called Local Linear Regression (LLR) to build a state-transition model during the learning process. It is expected that the quality of the model increases as the number of observed state-transitions increase. To assess the quality of the modeled state-transitions we introduce prediction intervals. We show that LLR is able to model various systems, including a complex humanoid robot. The LLR model was added to the learning algorithm to generate more state-transitions for the agent to learn from. We show that an increasing number of experiences leads to faster learning. We introduce Prioritized Sweeping (PS) and Look Ahead (LA) Dyna as possibilities to use the model more efficiently. We show how prediction intervals can be used to increase the performance of the various algorithms. The learning algorithms were compared using an inverted pendulum simulation, which had to learn a swing-up control task.

Reinforcement learning (RL) is one of the most active research areas in artificial intelligence. In RL an agent tries to maximize the total amount of reward it receives while interacting with an environment. The reward is used to improve the policy. Conventional methods of reinforcement learning perform well for simple tasks, but as the task becomes more complex, these methods fail to converge fast or converge to a suboptimal policy. Hence, new methods of RL are needed that can handle complex tasks. As humans, we simplify a task that is difficult to learn by first learning simplified versions of the task, before moving back to the original task. This idea of starting from simpler tasks and gradually increasing complexity, until the original task is solved, can also be exploited in RL, where it is called shaping. In order to accelerate learning in the original task, shaping methods transfer the knowledge and experiences from the easy task (source task) to the original task (target task). It is believed that the process of gradually increasing the complexity significantly reduces the difficulty of the learning problem. However, sometimes the total required time to solve the easy task plus the original task is larger than starting from scratch. In order to reduce this time, an essential decision in shaping is when to transfer learning from an easier task to a more difficult one. Transferring too early may mean the controller has not learned enough in the easy task, diminishing the usefulness of shaping. Transferring too late could make the controller waste learning time in the easy task, without making significant progress toward solving the original, complex task. If we switch the task at a proper point then the total learning time will decrease. The first part of this thesis is devoted to a thorough empirical study of the shaping methods. In the next part we try to find a suitable performance index that can be used as a switching criterion to reduce the total time needed to learn a task.

The advanced technology of today has reduced the need of testing medical equipment on human patients. The invention of the lung simulator has made it possible to test ventilators without putting the condition of test patients in jeopardy. The device also allows medical staff to be trained in a safe and more rapid way. Several types of lung simulators are available. The digitally controlled variant is the most versatile one for simulations. This type has easily adjustable static and dynamic properties, which, in contrary to traditional passive mechanical simulators, make it possible to simulate different types of breathing. The available design of a digitally controlled lung simulator, however, has one major drawback. The system uses a motor-ball screw assembly to drive a piston in the air compartment. One of the properties of this assembly is that the motor shaft motion is negligibly affected (not ‘backdrivable’) by air pressure exerted on the piston by an external source. The result of this is a feedback system that needs to realize the entire dynamic response to pressure changes, by active control of the motor. Because of this, the system can easily become unstable for certain settings of the static and dynamic properties. The objective of this thesis is to determine what the best alternative actuator is for the lung simulator, to the end of making it more stable than the current version. By improving the existing lung simulator and improving its range of operation, and its performance, it can be made more effective and accurate, which will result in better healthcare. To reach this goal, firstly the desired requirements for the lung simulator will be obtained from the end user. After this, a literature study about alternative actuators shall be conducted. With the results of the literature study, a ranking of the alternative actuators are determined. From the literature study eight alternative actuators have come up. These are the moving coil actuator, moving iron actuator, permanent magnet synchronous motor (PMSM), ironless core motor, piezoelectric actuator, pneumatic actuator, hydraulic actuator and wax motor. The most important requirements on which the different actuators are evaluated are backdrivability, force linearity, force, precision, speed and stroke length. Of the eight alternative actuators, except for the PMSM and the ironless core actuator, all the others have an insufficient score on one or more of the requirements. A comparison of the PMSM and the ironless core actuator shows that the backdrivability of the ironless core actuator is better, and that it has the best overall score. The conclusion is that the ironless core actuator is the most suitable alternative for the lung simulator. The use of the actuator should increase the stability of the lung simulator. However, no testing was possible because of malfunctioning hardware and time shortage. Therefore no practical confirmation could be obtained about the effects of the new actuator on the system. It is recommended to put the top-ranked actuators to the test in a prototype. From the test results it will be possible to confirm whether the instability of the original lung simulator was indeed caused by the actuator, and whether the ironless core motor is indeed the best suitable alternative actuator for the lung simulator.

The reinforcement learning (RL) framework enables to construct controllers that try to find find an optimal control strategy in an unknown environment by trial-and-error. After selecting a control action, the controller receives a numerical reward. The reward signal is based on the current state of the environment and the applied control action. The controller aims to maximize the cumulative reward, known as the return. In this thesis actor-critic and criticonly RL algorithms are considered. Actor-critic algorithms consist of an element that selects the actions (the actor) and an element that learns the expectation of the return (the critic). This expectation is captured in a value function. The critic is used to improve the control policy of the actor. Critic-only algorithms select the action by direct optimization over a value function. Before a RL algorithm can be applied to a control problem, a number of learning parameters need to be set. The optimal values of some of these parameters are highly problem dependent. It is not straightforward how these parameters should be chosen and often these are often determined by trying a large set of parameters. The main focus of thesis is to devise an action selection method that is able to select continuous actions without problem dependent parameters. Two approaches are taken: first, it is investigated if Levenberg Marquardt (LM), a popular optimization method, can be used to determine the actor update step. Second, an action selection method is treated that lacks an explicit actor, called Value-Gradient Based Policy (VGBP). The LM algorithm uses the gradient and the Hessian to compute the update step. Therefore the policy gradient and Hessian need to be found. A novel actor-critic method has been devised, called Vanilla Actor-Critic (VAC), that efficiently learns the policy gradient. On the inverted pendulum swing-up task this algorithm outperformed Natural Actor-Critic (NAC). A number of different approaches have been taken to approximate the policy Hessian, but none delivered a proper Hessian estimate. Therefore, no LM actor update method was created. In VGBP the action is found by optimization of the right hand side of the Bellman equation. VGBP uses the provided reward function and a process model for this optimization. The process model is learned online using local linear regression (LLR). Due to the efficient use of information VGBP shows fast learning on the pendulum and a 2-DOF robotic arm.

Mobile robots that communicate and cooperate to achieve a common task have been the subject of an increasing research interest in recent years. These possibly heterogeneous groups of robots communicate locally via a communication network and therefore are usually referred to as robotic networks. Their potential applications are diverse and encompass monitoring, exploration, search and rescue, and disaster relief. From a research standpoint, in this thesis we consider specific aspects related to the foundations of robotic network algorithmic development: distributed estimation, control, and optimization. The word “distributed” refers to situations in which the cooperating robots have a limited, local knowledge of the environment and of the group, as opposed to a “centralized” scenario, where all the robots have access to the complete information. The typical challenge in distributed systems is to achieve similar results (in terms of performance of the estimation, control, or optimization task) with respect to a centralized system without extensive communication among the cooperating robots. In this thesis we develop effective distributed estimation, control, and optimization algorithms tailored to the distributed nature of robotic networks. These algorithms strive for limiting the local communication among the mobile robots, in order to be applicable in practical situations. In particular, we focus on issues related to nonlinearities of the dynamical model of the robots and their sensors, to the connectivity of the communication graph through which the robots interact, and to fast feasible solutions for the common (estimation or control) objective.

Confidential report. Only a part of the thesis is presented in the repository.

The very basis of this thesis is the collective behavior of ants in colonies. Ants are an excellent example of how rather simple behavior on a local level can lead to complex behavior on a global level that is beneficial for the individuals. The key in the self-organization of ants is communication through pheromones. When an ant forages for food, it is biased to search along trails of stronger pheromone concentrations. The moment it finds food, it will walk back to the nest while depositing pheromones and thereby contributing to the reinforcement of a successful trail. Inspired by this mechanism, research within an engineering context has led to the development of the field of Ant Colony Optimization (ACO). Specifically developed for efficiently solving combinatorial optimization problems, ACO has been successfully applied to routing in road traffic and Internet networks. In this thesis, we take the principles behind ACO to the domain of control policy learning. A control policy is a mapping from states to actions and our objective is to develop methods to learn the optimal control policy for a given dynamic system by interacting with it. We call our methods Ant Colony Learning (ACL) and their power lies in the fact that there is a set of ants, from which each ant interacts with the system and influences the other ants through updating pheromone levels associated with the visited state-action pairs. In experiments involving control problems that have a discrete state space and deterministic state transitions, it turns out that ACL converges quickly to the optimal solution. We also observe that increasing the number of ants in the algorithm results in a decrease of the number of trials required for convergence to the optimal policy. An analytical study of the convergence behavior of ACL reveals that for systems with discrete and noiseless state transitions, the expected policy converges to the optimal policy in the case of using only one ant. Another major part of this thesis deals with the application of ACL to control problems with continuous state spaces. In order to capture a continuous space with a finite number of elements, we study two ways of partitioning the state space and their incorporation in the ACL framework. In crisp ACL, the state space is partitioned using bins. Each state measurement is assigned to exactly one bin, which leads to the introduction of discretization noise, rendering an originally deterministic system non-deterministic and restricting the performance of the algorithm. We find that a better way of partitioning the state space is by using fuzzy triangular membership functions. The continuous state measurement then belongs to multiple membership functions to a certain degree. With fuzzy partitioning, the continuity of the state variables is preserved and no non-determinism is introduced. We call this method fuzzy ACL. The developed generalized ACL algorithm unifies both crisp and fuzzy ACL. The behavior and performance of crisp and fuzzy ACL are further studied using simulation experiments. We study the influence of the local and global pheromone decay rates, the number of ants, and the density of the state space partitioning grid on the learning performance. Especially, the performance of crisp ACL improves for a small local pheromone decay rate, while fuzzy ACL outperforms crisp ACL over the whole line. In general, crisp ACL is much more sensitive to the choice of the pheromone decay parameters than fuzzy ACL. We find that using more ants leads to faster convergence, but that the number of ants does not need to be extremely large to obtain a satisfactory performance. With regard to the scaling of ACL, crisp ACL reveals a slow, but gradual improvement of the learning for an increasing state space partitioning density. Fuzzy ACL, on the other hand, improves more rapidly and requires fewer ants to learn a better control policy. Finally, we present a general modeling framework for swarms of moving agents. It turns out that ACL fits within this framework and as such can be unified with other swarm intelligence techniques. In the future, this could result in beneficial integration of elements from other swarm intelligence techniques into ACL, or the other way around.

Service robots have the potential to be of great value in households, health care and other labor intensive environments. However, these environments are typically unique, not very structured and frequently changing, which makes it difficult to make service robots robust and versatile through manual programming. Having robots learn to solve tasks autonomously through interaction with the real world forms an attractive alternative. With Reinforcement Learning (RL), a system can learn to perform tasks by receiving only coarse feedback on its actions: desired behavior is reinforced by positive rewards, undesired behavior is punished by negative rewards. In this research, a bipedal walking robot named Leo was designed and built specifically to study the application of RL to real robots. Robot Leo is able to learn two basic motor control tasks: placing a foot on a step of stairs, and walking. To learn to walk, Leo receives a positive reward for moving its foot forward, and negative rewards for falling and for spending time and energy. This process takes about 5 hours of practice in simulation, as well as thousands of falls. On the real prototype, the learning time was shortened by first letting the robot observe a hand coded, sub-optimal controller, which it was quickly able to mimic and even improve in a matter of hours. Algorithmic improvements are proposed to address complications of RL on real robots, such as time delays in the control loop and large disturbances such as a sudden push. To reduce the continuous risk of damage due to the trial-and-error nature of RL, a modular approach is proposed through which the robot can coarsely but quickly learn about the risk of its behavior and learn the actual task more safely and in more detail.

Adaptive fuzzy observers have been introduced in the recent past, which are capable of estimating uncertainties along with the states of a nonlinear system represented by an uncertain Takagi-Sugeno (TS) model. Application of such an observer to obtain estimates of the uncertainties in the state matrices and subsequently use them in the control of TS fuzzy models is the subject matter of this thesis. To demonstrate the adaptive observer and the controller design we use a 2-DOF robot arm model. The parameters of the robot arm are estimated for a laboratory-scale setup. The nonlinear model of the robot arm, consisting of six nonlinearities is simplified to contain only two nonlinearities. Then a four-rule TS fuzzy model is constructed using sector nonlinearity approach. The simplified nonlinear model and hence the fuzzy model almost exactly represent the complete nonlinear model. The mismatch in the plant and the nonlinear model is attributed to unmodelled dynamics in the state matrices. Assuming constant uncertainties in specific locations of the state matrices, an adaptive observer is used to estimate them and the simulation results are presented. The possibility to use the information about the structure of the uncertainties in the TS fuzzy model in designing the uncertainty estimation experiment is also presented. The uncertainty estimates provided by the adaptive observer are used to update the fuzzy model of the nonlinear system. The new model is used in the design of a robust state feedback stabilizing controller. Since the estimates obtained from the adaptive observer are used in controller design, the uncertainty distribution structure used in the design of both adaptive observer and the robust controller need to be same. Hence, a robust controller design is developed that uses the same uncertainty distribution structure as the adaptive observer. From the experimental results, it is concluded that stability can be guaranteed with a higher decay rate when using the updated model in robust controller design.