WP
W. Pan
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
14 records found
1
Towards Corrective Deep Imitation Learning in Data Intensive Environments
Helping robots to learn faster by leveraging human knowledge
Master thesis
(2021)
-
I. Lopez Bosque, Carlos Celemin Paez, Rodrigo Perez Dattari, Jens Kober, Wei Pan
Interactive imitation learning refers to learning methods where a human teacher interacts with an agent during the learning process providing feedback to improve its behaviour. This type of learning may be preferable with respect to reinforcement learning techniques when dealing with real-world problems. This fact is especially true in the case of robotic applications where reinforcement learning may be unfeasible as there are long training times and reward functions can be hard to shape/compute.
The present thesis focuses on interactive learning with corrective feedback and, in particular, in the framework Deep Corrective Advice Communicated by Humans (D-COACH), which has successfully shown to be advantageous in terms of training time and data efficiency. D-COACH, a supervised learning method whose policy is represented by an artificial neural network, incorporates a replay buffer where samples of states and corresponding labels gathered by the agent's policy from human feedback are stored and replayed. However, this causes conflicts between the data in the buffer because samples collected by older versions of the policy may be contradictory and could deteriorate the performance of the current policy. In order to reduce this issue, the current implementation of D-COACH uses a first-in-first-out buffer with limited size, as the older the sample is, the more likely it is to deteriorate the performance of the learner. Nonetheless, this limitation propitiates catastrophic forgetting, an inherent tendency of neural networks to forget what they have already learnt, and that can be mitigated by replaying information gathered during all the stages of the problem. Therefore, D-COACH suffers from a trade-off between reducing conflicting data and avoiding catastrophic forgetting. The fact that D-COACH limits the size of its buffer automatically restricts the types of problems that it can solve, given that, if the problem is too complex (i.e. it requires large amounts of data), it simply will not be able to remember everything.
If we want to utilise a buffer to train data intensive tasks with corrective feedback, a new method is needed to solve the problem of using information gathered by older versions of the policy. We propose an improved version of D-COACH, which we call Batch Deep COACH (BD-COACH, pronounced “be the coach”). BD-COACH incorporates a human model module that learns the feedback from the teacher and that can be employed to make corrections gathered by older versions of the policy still useful for batch updating the current version of the policy.
To compare the performance of BD-COACH with respect to D-COACH, three simulated experiments were done using the open-source Meta-World benchmark, which is based on MuJoCo and OpenAI gym. Moreover, to validate the proposed method in a real setup, two planar manipulation tasks were solved using a seven degrees of freedom KUKA robot arm.
Furthermore, we present an analysis between on-policy and off-policy methods both in the fields of reinforcement learning and in imitation learning. We believe there is an interesting simile between this classification and the problem of correctly implementing a replay buffer when learning from corrective feedback.
...
The present thesis focuses on interactive learning with corrective feedback and, in particular, in the framework Deep Corrective Advice Communicated by Humans (D-COACH), which has successfully shown to be advantageous in terms of training time and data efficiency. D-COACH, a supervised learning method whose policy is represented by an artificial neural network, incorporates a replay buffer where samples of states and corresponding labels gathered by the agent's policy from human feedback are stored and replayed. However, this causes conflicts between the data in the buffer because samples collected by older versions of the policy may be contradictory and could deteriorate the performance of the current policy. In order to reduce this issue, the current implementation of D-COACH uses a first-in-first-out buffer with limited size, as the older the sample is, the more likely it is to deteriorate the performance of the learner. Nonetheless, this limitation propitiates catastrophic forgetting, an inherent tendency of neural networks to forget what they have already learnt, and that can be mitigated by replaying information gathered during all the stages of the problem. Therefore, D-COACH suffers from a trade-off between reducing conflicting data and avoiding catastrophic forgetting. The fact that D-COACH limits the size of its buffer automatically restricts the types of problems that it can solve, given that, if the problem is too complex (i.e. it requires large amounts of data), it simply will not be able to remember everything.
If we want to utilise a buffer to train data intensive tasks with corrective feedback, a new method is needed to solve the problem of using information gathered by older versions of the policy. We propose an improved version of D-COACH, which we call Batch Deep COACH (BD-COACH, pronounced “be the coach”). BD-COACH incorporates a human model module that learns the feedback from the teacher and that can be employed to make corrections gathered by older versions of the policy still useful for batch updating the current version of the policy.
To compare the performance of BD-COACH with respect to D-COACH, three simulated experiments were done using the open-source Meta-World benchmark, which is based on MuJoCo and OpenAI gym. Moreover, to validate the proposed method in a real setup, two planar manipulation tasks were solved using a seven degrees of freedom KUKA robot arm.
Furthermore, we present an analysis between on-policy and off-policy methods both in the fields of reinforcement learning and in imitation learning. We believe there is an interesting simile between this classification and the problem of correctly implementing a replay buffer when learning from corrective feedback.
...
Interactive imitation learning refers to learning methods where a human teacher interacts with an agent during the learning process providing feedback to improve its behaviour. This type of learning may be preferable with respect to reinforcement learning techniques when dealing with real-world problems. This fact is especially true in the case of robotic applications where reinforcement learning may be unfeasible as there are long training times and reward functions can be hard to shape/compute.
The present thesis focuses on interactive learning with corrective feedback and, in particular, in the framework Deep Corrective Advice Communicated by Humans (D-COACH), which has successfully shown to be advantageous in terms of training time and data efficiency. D-COACH, a supervised learning method whose policy is represented by an artificial neural network, incorporates a replay buffer where samples of states and corresponding labels gathered by the agent's policy from human feedback are stored and replayed. However, this causes conflicts between the data in the buffer because samples collected by older versions of the policy may be contradictory and could deteriorate the performance of the current policy. In order to reduce this issue, the current implementation of D-COACH uses a first-in-first-out buffer with limited size, as the older the sample is, the more likely it is to deteriorate the performance of the learner. Nonetheless, this limitation propitiates catastrophic forgetting, an inherent tendency of neural networks to forget what they have already learnt, and that can be mitigated by replaying information gathered during all the stages of the problem. Therefore, D-COACH suffers from a trade-off between reducing conflicting data and avoiding catastrophic forgetting. The fact that D-COACH limits the size of its buffer automatically restricts the types of problems that it can solve, given that, if the problem is too complex (i.e. it requires large amounts of data), it simply will not be able to remember everything.
If we want to utilise a buffer to train data intensive tasks with corrective feedback, a new method is needed to solve the problem of using information gathered by older versions of the policy. We propose an improved version of D-COACH, which we call Batch Deep COACH (BD-COACH, pronounced “be the coach”). BD-COACH incorporates a human model module that learns the feedback from the teacher and that can be employed to make corrections gathered by older versions of the policy still useful for batch updating the current version of the policy.
To compare the performance of BD-COACH with respect to D-COACH, three simulated experiments were done using the open-source Meta-World benchmark, which is based on MuJoCo and OpenAI gym. Moreover, to validate the proposed method in a real setup, two planar manipulation tasks were solved using a seven degrees of freedom KUKA robot arm.
Furthermore, we present an analysis between on-policy and off-policy methods both in the fields of reinforcement learning and in imitation learning. We believe there is an interesting simile between this classification and the problem of correctly implementing a replay buffer when learning from corrective feedback.
The present thesis focuses on interactive learning with corrective feedback and, in particular, in the framework Deep Corrective Advice Communicated by Humans (D-COACH), which has successfully shown to be advantageous in terms of training time and data efficiency. D-COACH, a supervised learning method whose policy is represented by an artificial neural network, incorporates a replay buffer where samples of states and corresponding labels gathered by the agent's policy from human feedback are stored and replayed. However, this causes conflicts between the data in the buffer because samples collected by older versions of the policy may be contradictory and could deteriorate the performance of the current policy. In order to reduce this issue, the current implementation of D-COACH uses a first-in-first-out buffer with limited size, as the older the sample is, the more likely it is to deteriorate the performance of the learner. Nonetheless, this limitation propitiates catastrophic forgetting, an inherent tendency of neural networks to forget what they have already learnt, and that can be mitigated by replaying information gathered during all the stages of the problem. Therefore, D-COACH suffers from a trade-off between reducing conflicting data and avoiding catastrophic forgetting. The fact that D-COACH limits the size of its buffer automatically restricts the types of problems that it can solve, given that, if the problem is too complex (i.e. it requires large amounts of data), it simply will not be able to remember everything.
If we want to utilise a buffer to train data intensive tasks with corrective feedback, a new method is needed to solve the problem of using information gathered by older versions of the policy. We propose an improved version of D-COACH, which we call Batch Deep COACH (BD-COACH, pronounced “be the coach”). BD-COACH incorporates a human model module that learns the feedback from the teacher and that can be employed to make corrections gathered by older versions of the policy still useful for batch updating the current version of the policy.
To compare the performance of BD-COACH with respect to D-COACH, three simulated experiments were done using the open-source Meta-World benchmark, which is based on MuJoCo and OpenAI gym. Moreover, to validate the proposed method in a real setup, two planar manipulation tasks were solved using a seven degrees of freedom KUKA robot arm.
Furthermore, we present an analysis between on-policy and off-policy methods both in the fields of reinforcement learning and in imitation learning. We believe there is an interesting simile between this classification and the problem of correctly implementing a replay buffer when learning from corrective feedback.
In this project, a unique method of combining online learning with model predictive control is applied to autonomous racing. A concern in autonomous racing is that accurate models that encapsulate the dynamics of the vehicle are complex, nonlinear, and difficult to identify. In order to make this more practical for control purposes, the controller is initialized with a nominal tire model, which then learns tire properties online using locally weighted projection regression during the course of the race. This makes it more practical for control purposes while maintaining model accuracy. Focus is placed on learning the tire properties which in reality, keep varying due to wear, temperature and pressure fluctuations, etc. The main objective is to minimize lap times by allowing the controller to "learn" its varying tire behavior while on the track.
...
In this project, a unique method of combining online learning with model predictive control is applied to autonomous racing. A concern in autonomous racing is that accurate models that encapsulate the dynamics of the vehicle are complex, nonlinear, and difficult to identify. In order to make this more practical for control purposes, the controller is initialized with a nominal tire model, which then learns tire properties online using locally weighted projection regression during the course of the race. This makes it more practical for control purposes while maintaining model accuracy. Focus is placed on learning the tire properties which in reality, keep varying due to wear, temperature and pressure fluctuations, etc. The main objective is to minimize lap times by allowing the controller to "learn" its varying tire behavior while on the track.
Interactive machine learning describes a collection of methodologies in which a human user actively participates in a novice agent’s learning process, through providing corrective or evaluate feedback or demonstrative actions. A primary assumption in these methods is that user input is at worst nearoptimal, however a realistic set of demonstrations will often contain conflicting or poor examples, which degrade the quality of the learnt policies. This project explores methods for the detection of such undesirable features in data and develops an algorithm for policy training with suboptimal demonstrations, while leveraging the generalisation and scalability qualities of artificial neural networks. Uncertainty estimation, which presents a structured approach for the quantification of a network’s confidence in the accuracy of its output, based on the observed training data, is applied for the detection of unwanted features in a demonstration dataset. The particular focus of this project is conflicting data resulting from scenarios with equivalent action choices, such as the obstacle avoidance setting. Following thorough testing on various environments, it is shown that novice policies may be trained to achieve a desired goal in multi-dimensional spaces with either discrete or continuous data, despite the presence of conflicts in this training data.
...
Interactive machine learning describes a collection of methodologies in which a human user actively participates in a novice agent’s learning process, through providing corrective or evaluate feedback or demonstrative actions. A primary assumption in these methods is that user input is at worst nearoptimal, however a realistic set of demonstrations will often contain conflicting or poor examples, which degrade the quality of the learnt policies. This project explores methods for the detection of such undesirable features in data and develops an algorithm for policy training with suboptimal demonstrations, while leveraging the generalisation and scalability qualities of artificial neural networks. Uncertainty estimation, which presents a structured approach for the quantification of a network’s confidence in the accuracy of its output, based on the observed training data, is applied for the detection of unwanted features in a demonstration dataset. The particular focus of this project is conflicting data resulting from scenarios with equivalent action choices, such as the obstacle avoidance setting. Following thorough testing on various environments, it is shown that novice policies may be trained to achieve a desired goal in multi-dimensional spaces with either discrete or continuous data, despite the presence of conflicts in this training data.
The use of artificial neural networks is becoming ever more ubiquitous as the computational power available to use grows. The widespread implementation of neural networks as controllers in the field of systems and control is however being hindered by the lack of verifiability of these controllers. One type of controller that does not lack verifiability is the correct-by-design controller. The main drawback of correct-by-design controllers is that they inherently produce large data structures in order to store their control rules. In this work, a novel methodology to synthesize correct-by-design neural network controllers is presented in order to alleviate these issues. This methodology combines reinforcement learning techniques and abstraction based correct-by-design control verification techniques in order to synthesize neural network controllers that are correct-by-design. The procedure does so by alternating between a controller training routine and a controller verification routine in a system abstraction framework. This framework ensures that numerical training and verification results in a controller with formal guarantees applicable to real systems. Using the proposed methodology, neural network controllers are synthesized and verified in order to prove that the methodology works. In addition to this, the resulting correct-by-design neural network controllers are compared to conventional correct-by-design controllers in order to judge their performance and data requirements. This comparison also includes alternative structures used to store these different controllers. Based on this comparison, a conclusion is drawn regarding when to use which type of controller. The proposed methodology is implemented into a correct-by-design neural network synthesis framework called COSYNNC. This framework is intended as a basis for further research into correct-by-design neural network control and is publicly available.
...
The use of artificial neural networks is becoming ever more ubiquitous as the computational power available to use grows. The widespread implementation of neural networks as controllers in the field of systems and control is however being hindered by the lack of verifiability of these controllers. One type of controller that does not lack verifiability is the correct-by-design controller. The main drawback of correct-by-design controllers is that they inherently produce large data structures in order to store their control rules. In this work, a novel methodology to synthesize correct-by-design neural network controllers is presented in order to alleviate these issues. This methodology combines reinforcement learning techniques and abstraction based correct-by-design control verification techniques in order to synthesize neural network controllers that are correct-by-design. The procedure does so by alternating between a controller training routine and a controller verification routine in a system abstraction framework. This framework ensures that numerical training and verification results in a controller with formal guarantees applicable to real systems. Using the proposed methodology, neural network controllers are synthesized and verified in order to prove that the methodology works. In addition to this, the resulting correct-by-design neural network controllers are compared to conventional correct-by-design controllers in order to judge their performance and data requirements. This comparison also includes alternative structures used to store these different controllers. Based on this comparison, a conclusion is drawn regarding when to use which type of controller. The proposed methodology is implemented into a correct-by-design neural network synthesis framework called COSYNNC. This framework is intended as a basis for further research into correct-by-design neural network control and is publicly available.
Deep Reinforcement Learning (DRL) enables us to design controllers for complex tasks with a deep learning approach. It allows us to design controllers that are otherwise cumbersome to design with conventional control methodologies. Often, an objective for RL is binary in nature. However, exploring in environments with sparse rewards is a problem in RL, and finding positive reward becomes exponentially more difficult with increased environment complexity. For this project, our objective is to design an RL based controller for the landing of a quadcopter on inclined surfaces. Landing is defined as reaching these inclined surfaces with reasonable speed, such that no damage is done to either the quadcopter or the surface to land on upon impact. We aim to use a binary reward for this task. We use methods to aid exploration in sparse reward environments, namely Hindsight Experience Replay (HER), and non-optimized demonstrations. HER can resample goals from the demonstrator data and the policy rollouts. The resampling of goals is done by considering a portion of the visited states during policy rollouts as the intended goals. The demonstrations are non-optimized in the sense that the demonstrations do not follow the same objective as ours. We consider demonstrations valid if these demonstrations are obtained from arbitrary stable policies. Our results show that the RL system does generalize to other goals when using HER and demonstrations. The demonstrations are not imitated as were to happen in pure imitation learning. HER, on the other hand, enabled us to receive reward in our complex environment, while also allowing us to experience multiple goals in one policy rollout. We found that lack of HER and demonstrations were not able to overcome the problems of exploration in sparse reward environments. We found that landing a quadcopter on inclined surfaces using an RL controller is feasible. Our trajectories clearly showed a swinging motion which in theory should be a valid control strategy for this problem. This swinging motion results in dead spots with the quadcopter being in a state with a minimal translational and rotational velocities under a relatively large angle. Further research is needed to increase the accuracy and robustness of our RL based controller.
...
Deep Reinforcement Learning (DRL) enables us to design controllers for complex tasks with a deep learning approach. It allows us to design controllers that are otherwise cumbersome to design with conventional control methodologies. Often, an objective for RL is binary in nature. However, exploring in environments with sparse rewards is a problem in RL, and finding positive reward becomes exponentially more difficult with increased environment complexity. For this project, our objective is to design an RL based controller for the landing of a quadcopter on inclined surfaces. Landing is defined as reaching these inclined surfaces with reasonable speed, such that no damage is done to either the quadcopter or the surface to land on upon impact. We aim to use a binary reward for this task. We use methods to aid exploration in sparse reward environments, namely Hindsight Experience Replay (HER), and non-optimized demonstrations. HER can resample goals from the demonstrator data and the policy rollouts. The resampling of goals is done by considering a portion of the visited states during policy rollouts as the intended goals. The demonstrations are non-optimized in the sense that the demonstrations do not follow the same objective as ours. We consider demonstrations valid if these demonstrations are obtained from arbitrary stable policies. Our results show that the RL system does generalize to other goals when using HER and demonstrations. The demonstrations are not imitated as were to happen in pure imitation learning. HER, on the other hand, enabled us to receive reward in our complex environment, while also allowing us to experience multiple goals in one policy rollout. We found that lack of HER and demonstrations were not able to overcome the problems of exploration in sparse reward environments. We found that landing a quadcopter on inclined surfaces using an RL controller is feasible. Our trajectories clearly showed a swinging motion which in theory should be a valid control strategy for this problem. This swinging motion results in dead spots with the quadcopter being in a state with a minimal translational and rotational velocities under a relatively large angle. Further research is needed to increase the accuracy and robustness of our RL based controller.
This master thesis presents an experimental study on 3D person localization (i.e., pedestrians, cyclists)in traffic scenes, using monocular vision and Light Detection And Ranging (LiDAR) data. The performance of two top-ranking methods is analyzed on the 3D object detection KITTI dataset. In this evaluation, the effect of the Intersection over Union (IoU) threshold on the performance in terms of 3D bounding box location, size, and orientation is analysed.
Since the KITTI 3D object detection dataset contains relatively few 3D person instances, the analysis will is to the EuroCity Persons 2.5D (ECP2.5D) datasets (both day and night), which is one order of magnitude larger. Using both datasets, additional experiments are performed to evaluate the influence of distance, the number of LiDAR points, occlusion, and intensity on the performance. Domain transfer experiments between the KITTI and ECP2.5D datasets are performed, to examine how these datasets generalize with respect to each other. Furthermore, Part-A2 net is used to evaluate the detection score which is given to the ground truth pedestrians. The relationship between the detection score and the distance, the number of LiDAR points, and occlusion is analyzed. Some objects are not detected although their ground truth detection score is high. This creates the potential to detect these pedestrians. Lastly, this thesis presents a method that uses the detections from the previous frame to increase the performance in the subsequent frame by adding the previous detections to the 3D proposals coming from the Region Proposal Network (RPN). ...
Since the KITTI 3D object detection dataset contains relatively few 3D person instances, the analysis will is to the EuroCity Persons 2.5D (ECP2.5D) datasets (both day and night), which is one order of magnitude larger. Using both datasets, additional experiments are performed to evaluate the influence of distance, the number of LiDAR points, occlusion, and intensity on the performance. Domain transfer experiments between the KITTI and ECP2.5D datasets are performed, to examine how these datasets generalize with respect to each other. Furthermore, Part-A2 net is used to evaluate the detection score which is given to the ground truth pedestrians. The relationship between the detection score and the distance, the number of LiDAR points, and occlusion is analyzed. Some objects are not detected although their ground truth detection score is high. This creates the potential to detect these pedestrians. Lastly, this thesis presents a method that uses the detections from the previous frame to increase the performance in the subsequent frame by adding the previous detections to the 3D proposals coming from the Region Proposal Network (RPN). ...
This master thesis presents an experimental study on 3D person localization (i.e., pedestrians, cyclists)in traffic scenes, using monocular vision and Light Detection And Ranging (LiDAR) data. The performance of two top-ranking methods is analyzed on the 3D object detection KITTI dataset. In this evaluation, the effect of the Intersection over Union (IoU) threshold on the performance in terms of 3D bounding box location, size, and orientation is analysed.
Since the KITTI 3D object detection dataset contains relatively few 3D person instances, the analysis will is to the EuroCity Persons 2.5D (ECP2.5D) datasets (both day and night), which is one order of magnitude larger. Using both datasets, additional experiments are performed to evaluate the influence of distance, the number of LiDAR points, occlusion, and intensity on the performance. Domain transfer experiments between the KITTI and ECP2.5D datasets are performed, to examine how these datasets generalize with respect to each other. Furthermore, Part-A2 net is used to evaluate the detection score which is given to the ground truth pedestrians. The relationship between the detection score and the distance, the number of LiDAR points, and occlusion is analyzed. Some objects are not detected although their ground truth detection score is high. This creates the potential to detect these pedestrians. Lastly, this thesis presents a method that uses the detections from the previous frame to increase the performance in the subsequent frame by adding the previous detections to the 3D proposals coming from the Region Proposal Network (RPN).
Since the KITTI 3D object detection dataset contains relatively few 3D person instances, the analysis will is to the EuroCity Persons 2.5D (ECP2.5D) datasets (both day and night), which is one order of magnitude larger. Using both datasets, additional experiments are performed to evaluate the influence of distance, the number of LiDAR points, occlusion, and intensity on the performance. Domain transfer experiments between the KITTI and ECP2.5D datasets are performed, to examine how these datasets generalize with respect to each other. Furthermore, Part-A2 net is used to evaluate the detection score which is given to the ground truth pedestrians. The relationship between the detection score and the distance, the number of LiDAR points, and occlusion is analyzed. Some objects are not detected although their ground truth detection score is high. This creates the potential to detect these pedestrians. Lastly, this thesis presents a method that uses the detections from the previous frame to increase the performance in the subsequent frame by adding the previous detections to the 3D proposals coming from the Region Proposal Network (RPN).
Embedded control systems are processor-based systems that need to run an application for an extended amount of time, such as months or years. Typically, they implement a realtime function to control a system. Embedded systems are implemented using hardware and software to perform an specific task. This is why they can be optimized to reduce its size and cost and increase its reliability and performance. In embedded control systems, a discrete time embedded system is controlling a continuous time plant. In order to deal with this complex interactions, there are some tools that synthesize symbolic controllers. However,
the size of these controllers is still too large to be widely implemented in embedded systems for real-time applications. Although it is possible to implement them in CPUs with large memory, their time-step is limited by a few GHz. On the other hand, FPGAs can run at a higher frequency (MHz) but they have limited memory. In this project, we propose a tool that automate the process of compressing, determinizing and generating the necessary files to flash a symbolic controller into an FPGA. We propose three different ways of transforming the original controllers and we compare them with another similar tool from the Technische Universität München. We also simulate in real-time the controlled closed-loop of some of those symbolic controllers using a simulated plant to validate the entire process. ...
the size of these controllers is still too large to be widely implemented in embedded systems for real-time applications. Although it is possible to implement them in CPUs with large memory, their time-step is limited by a few GHz. On the other hand, FPGAs can run at a higher frequency (MHz) but they have limited memory. In this project, we propose a tool that automate the process of compressing, determinizing and generating the necessary files to flash a symbolic controller into an FPGA. We propose three different ways of transforming the original controllers and we compare them with another similar tool from the Technische Universität München. We also simulate in real-time the controlled closed-loop of some of those symbolic controllers using a simulated plant to validate the entire process. ...
Embedded control systems are processor-based systems that need to run an application for an extended amount of time, such as months or years. Typically, they implement a realtime function to control a system. Embedded systems are implemented using hardware and software to perform an specific task. This is why they can be optimized to reduce its size and cost and increase its reliability and performance. In embedded control systems, a discrete time embedded system is controlling a continuous time plant. In order to deal with this complex interactions, there are some tools that synthesize symbolic controllers. However,
the size of these controllers is still too large to be widely implemented in embedded systems for real-time applications. Although it is possible to implement them in CPUs with large memory, their time-step is limited by a few GHz. On the other hand, FPGAs can run at a higher frequency (MHz) but they have limited memory. In this project, we propose a tool that automate the process of compressing, determinizing and generating the necessary files to flash a symbolic controller into an FPGA. We propose three different ways of transforming the original controllers and we compare them with another similar tool from the Technische Universität München. We also simulate in real-time the controlled closed-loop of some of those symbolic controllers using a simulated plant to validate the entire process.
the size of these controllers is still too large to be widely implemented in embedded systems for real-time applications. Although it is possible to implement them in CPUs with large memory, their time-step is limited by a few GHz. On the other hand, FPGAs can run at a higher frequency (MHz) but they have limited memory. In this project, we propose a tool that automate the process of compressing, determinizing and generating the necessary files to flash a symbolic controller into an FPGA. We propose three different ways of transforming the original controllers and we compare them with another similar tool from the Technische Universität München. We also simulate in real-time the controlled closed-loop of some of those symbolic controllers using a simulated plant to validate the entire process.
In the past few years, convolutional neural networks (CNNs) have been widely utilized and shown state-of-the-art performances on computer vision tasks. However, CNN based approaches usually require a large amount of storage, run-time memory, as well as computation power in both training and inference time, which are usually used on GPU based machines to ensure the speed for inferences. But they are usually insufficient to be deployed on low-power applications. Although many approaches were proposed to compress and accelerate the CNN models, most of them were only evaluated on relatively simple problems (e.g. image classification), which only support limited real-world applications. Especially, among those methods, binary quantization can achieve very high model compression, but only a few works have been observed to utilize it on more complex tasks. Therefore, the exploration and evaluations of applying binary quantization on more complex tasks like object detection are worthwhile, which can be used in much more applications like autonomous driving and face detection. In this project, we apply and evaluate two different binary quantization approaches, named ABC-Net and PA-Net on object detection tasks. Also, we specify the exact implementation details for the binary convolutional operations in this project. As a result, we can achieve maximally 6.1× (around 16% of the full-precision model) compression, and minimal 2.5% accuracy reduction for weight quantization. The weight quantized models were able to outperform some existing real-time detectors in terms of both accuracy and storage size. Although large accuracy reduction was observed for input quantization, the quantized model could still maintain an acceptable accuracy compared to existing real-time object detectors.
...
In the past few years, convolutional neural networks (CNNs) have been widely utilized and shown state-of-the-art performances on computer vision tasks. However, CNN based approaches usually require a large amount of storage, run-time memory, as well as computation power in both training and inference time, which are usually used on GPU based machines to ensure the speed for inferences. But they are usually insufficient to be deployed on low-power applications. Although many approaches were proposed to compress and accelerate the CNN models, most of them were only evaluated on relatively simple problems (e.g. image classification), which only support limited real-world applications. Especially, among those methods, binary quantization can achieve very high model compression, but only a few works have been observed to utilize it on more complex tasks. Therefore, the exploration and evaluations of applying binary quantization on more complex tasks like object detection are worthwhile, which can be used in much more applications like autonomous driving and face detection. In this project, we apply and evaluate two different binary quantization approaches, named ABC-Net and PA-Net on object detection tasks. Also, we specify the exact implementation details for the binary convolutional operations in this project. As a result, we can achieve maximally 6.1× (around 16% of the full-precision model) compression, and minimal 2.5% accuracy reduction for weight quantization. The weight quantized models were able to outperform some existing real-time detectors in terms of both accuracy and storage size. Although large accuracy reduction was observed for input quantization, the quantized model could still maintain an acceptable accuracy compared to existing real-time object detectors.
Neural networks have achieved great success in many difficult learning tasks like image classification, speech recognition and natural language processing. However, neural architectures are hard to design, which requires lots of knowledge and time of human experts. Therefore, there has been a growing interest in automating the process of designing neural architectures. Though these searched architectures have achieved competitive performance on various tasks, the efficiency of NAS still needs to be improved. Moreover, current neural architecture search approach disregards the dependency between a node and its predecessors and successors.
This thesis builds upon BayesNAS which employs the classic Bayesian learning method to search for CNN architectures, and extends it to the problem of neural architecture search for recurrent architectures. Hierarchical sparse priors are used to model the architecture parameters to alleviate the dependency issue. Since the update of posterior variance is based on Laplace approximation, an efficient method to compute the Hessian of recurrent layer is proposed. We can find candidated architectures after training the over-parameterized network for only one epoch. Our experiments on Penn Treebank and WikiText-2 show that competitive architectures can be found in 0.3 GPU days using a single GPU for language modeling task. We find that our algorithm is more efficient than state-of-the-art.
...
This thesis builds upon BayesNAS which employs the classic Bayesian learning method to search for CNN architectures, and extends it to the problem of neural architecture search for recurrent architectures. Hierarchical sparse priors are used to model the architecture parameters to alleviate the dependency issue. Since the update of posterior variance is based on Laplace approximation, an efficient method to compute the Hessian of recurrent layer is proposed. We can find candidated architectures after training the over-parameterized network for only one epoch. Our experiments on Penn Treebank and WikiText-2 show that competitive architectures can be found in 0.3 GPU days using a single GPU for language modeling task. We find that our algorithm is more efficient than state-of-the-art.
...
Neural networks have achieved great success in many difficult learning tasks like image classification, speech recognition and natural language processing. However, neural architectures are hard to design, which requires lots of knowledge and time of human experts. Therefore, there has been a growing interest in automating the process of designing neural architectures. Though these searched architectures have achieved competitive performance on various tasks, the efficiency of NAS still needs to be improved. Moreover, current neural architecture search approach disregards the dependency between a node and its predecessors and successors.
This thesis builds upon BayesNAS which employs the classic Bayesian learning method to search for CNN architectures, and extends it to the problem of neural architecture search for recurrent architectures. Hierarchical sparse priors are used to model the architecture parameters to alleviate the dependency issue. Since the update of posterior variance is based on Laplace approximation, an efficient method to compute the Hessian of recurrent layer is proposed. We can find candidated architectures after training the over-parameterized network for only one epoch. Our experiments on Penn Treebank and WikiText-2 show that competitive architectures can be found in 0.3 GPU days using a single GPU for language modeling task. We find that our algorithm is more efficient than state-of-the-art.
This thesis builds upon BayesNAS which employs the classic Bayesian learning method to search for CNN architectures, and extends it to the problem of neural architecture search for recurrent architectures. Hierarchical sparse priors are used to model the architecture parameters to alleviate the dependency issue. Since the update of posterior variance is based on Laplace approximation, an efficient method to compute the Hessian of recurrent layer is proposed. We can find candidated architectures after training the over-parameterized network for only one epoch. Our experiments on Penn Treebank and WikiText-2 show that competitive architectures can be found in 0.3 GPU days using a single GPU for language modeling task. We find that our algorithm is more efficient than state-of-the-art.
Online Reinforcement Learning for Flight Control
An Adaptive Critic Design without prior model knowledge
Online Reinforcement Learning is a possible solution for adaptive nonlinear flight control. In this research an Adaptive Critic Design (ACD) based on Dual Heuristic Dynamic Programming (DHP) is developed and implemented on a simulated Cessna Citation 550 aircraft. Using an online identified system model approximation, the method is independent of prior model knowledge. The agent consists of two Artificial Neural Networks (ANNs) which form the Adaptive Critic Design and is supplemented with a Recursive Least Squares (RLS) online model estimation. The implemented agent is demonstrated to learn a near optimal control policy for different operating points, which is capable of tracking pitch and roll rate while actively minimizing the sideslip angle in a faster than real-time simulation. Providing limited model knowledge is shown to increase the learning, performance and robustness of the controller.
...
Online Reinforcement Learning is a possible solution for adaptive nonlinear flight control. In this research an Adaptive Critic Design (ACD) based on Dual Heuristic Dynamic Programming (DHP) is developed and implemented on a simulated Cessna Citation 550 aircraft. Using an online identified system model approximation, the method is independent of prior model knowledge. The agent consists of two Artificial Neural Networks (ANNs) which form the Adaptive Critic Design and is supplemented with a Recursive Least Squares (RLS) online model estimation. The implemented agent is demonstrated to learn a near optimal control policy for different operating points, which is capable of tracking pitch and roll rate while actively minimizing the sideslip angle in a faster than real-time simulation. Providing limited model knowledge is shown to increase the learning, performance and robustness of the controller.
Automated vehicles are conventional vehicles equipped with advanced sensors, controller and actuators. They achieve intelligent information exchange with the environment through the onboard sensing and cooperative system. vehicles are possible to have situation awareness and automatically analyze the safety and dangerous state of journeys. Finally vehicles can reach destinations following drivers' willing. The ongoing research on intelligent vehicles is mainly about improving the safety, comfort, efficiency and provide an excellent human-car interface. As a self-organizing system, the traffic system is quite complicated. There are many disturbance factors to lead to various traffic problems. One of the daily occurring problems is congestion on the motorway. In order to reduce congestion, Rijkswaterstaat applies various dynamic traffic management (DTM) measures to guide the traffic. It works well nowadays in conventional traffic. However, automated vehicles entered the market recently and will start to play an essential role in future traffic. The automated vehicles' reaction to DTM measures may be different from conventional vehicles while the traffic problems still exist. Therefore, it is necessary to research the effectiveness of current Dutch traffic management in automated vehicles. This thesis aims to investigate the effectiveness of current Dutch DTM measures with driver assistant and partially automated vehicles. Due to the time limitation, only the ramp metering measure will be researched through a simulation study. Therefore the main research question is 'How partial automated driving influences the performance of current Dutch dynamic traffic management system and how can this be evaluated via simulation?'. Three methods are applied, including literature review, simulation and statistical analysis. The literature part reviews levels of automation, various longitudinal and lateral vehicle motion models, which are chosen and modified in the simulation. Many ramp metering algorithms are also introduced in the literature review. The ramp metering controller in the simulation follows RWS algorithm. Besides, the motorway demand and the penetration rate of level 1 and 2 vehicles are two input of the simulation.
From the simulation results, it is concluded that the level 2 automation consisting of Adaptive Cruise Control (ACC) and Lane Change Assistance (LCA) system brings a negative impact on the motorway capacity. The ramp metering measure remains efficient if the penetration rate of level 2 vehicles is low. However, when the capacity reduces to the critical flow set up in the ramp metering controller, Ramp metering loses its efficiency. The parameters in the ramp metering controller therefore, require an update. For further research, it is recommended to simulate the same scenarios with different ramp metering algorithms. Since the functions of the algorithms are different, there might be other robust control algorithms for automated vehicles. Besides, another limitation of this thesis is that the automation system in level 2 vehicles is defined as Adaptive Cruise Control (ACC) plus Lane Change Assistance (LCA) system. Other partial automation systems may have a different effect on the performance of ramp metering. This thesis can be expanded by research the ramp metering performance under various types of partial automation systems. ...
From the simulation results, it is concluded that the level 2 automation consisting of Adaptive Cruise Control (ACC) and Lane Change Assistance (LCA) system brings a negative impact on the motorway capacity. The ramp metering measure remains efficient if the penetration rate of level 2 vehicles is low. However, when the capacity reduces to the critical flow set up in the ramp metering controller, Ramp metering loses its efficiency. The parameters in the ramp metering controller therefore, require an update. For further research, it is recommended to simulate the same scenarios with different ramp metering algorithms. Since the functions of the algorithms are different, there might be other robust control algorithms for automated vehicles. Besides, another limitation of this thesis is that the automation system in level 2 vehicles is defined as Adaptive Cruise Control (ACC) plus Lane Change Assistance (LCA) system. Other partial automation systems may have a different effect on the performance of ramp metering. This thesis can be expanded by research the ramp metering performance under various types of partial automation systems. ...
Automated vehicles are conventional vehicles equipped with advanced sensors, controller and actuators. They achieve intelligent information exchange with the environment through the onboard sensing and cooperative system. vehicles are possible to have situation awareness and automatically analyze the safety and dangerous state of journeys. Finally vehicles can reach destinations following drivers' willing. The ongoing research on intelligent vehicles is mainly about improving the safety, comfort, efficiency and provide an excellent human-car interface. As a self-organizing system, the traffic system is quite complicated. There are many disturbance factors to lead to various traffic problems. One of the daily occurring problems is congestion on the motorway. In order to reduce congestion, Rijkswaterstaat applies various dynamic traffic management (DTM) measures to guide the traffic. It works well nowadays in conventional traffic. However, automated vehicles entered the market recently and will start to play an essential role in future traffic. The automated vehicles' reaction to DTM measures may be different from conventional vehicles while the traffic problems still exist. Therefore, it is necessary to research the effectiveness of current Dutch traffic management in automated vehicles. This thesis aims to investigate the effectiveness of current Dutch DTM measures with driver assistant and partially automated vehicles. Due to the time limitation, only the ramp metering measure will be researched through a simulation study. Therefore the main research question is 'How partial automated driving influences the performance of current Dutch dynamic traffic management system and how can this be evaluated via simulation?'. Three methods are applied, including literature review, simulation and statistical analysis. The literature part reviews levels of automation, various longitudinal and lateral vehicle motion models, which are chosen and modified in the simulation. Many ramp metering algorithms are also introduced in the literature review. The ramp metering controller in the simulation follows RWS algorithm. Besides, the motorway demand and the penetration rate of level 1 and 2 vehicles are two input of the simulation.
From the simulation results, it is concluded that the level 2 automation consisting of Adaptive Cruise Control (ACC) and Lane Change Assistance (LCA) system brings a negative impact on the motorway capacity. The ramp metering measure remains efficient if the penetration rate of level 2 vehicles is low. However, when the capacity reduces to the critical flow set up in the ramp metering controller, Ramp metering loses its efficiency. The parameters in the ramp metering controller therefore, require an update. For further research, it is recommended to simulate the same scenarios with different ramp metering algorithms. Since the functions of the algorithms are different, there might be other robust control algorithms for automated vehicles. Besides, another limitation of this thesis is that the automation system in level 2 vehicles is defined as Adaptive Cruise Control (ACC) plus Lane Change Assistance (LCA) system. Other partial automation systems may have a different effect on the performance of ramp metering. This thesis can be expanded by research the ramp metering performance under various types of partial automation systems.
From the simulation results, it is concluded that the level 2 automation consisting of Adaptive Cruise Control (ACC) and Lane Change Assistance (LCA) system brings a negative impact on the motorway capacity. The ramp metering measure remains efficient if the penetration rate of level 2 vehicles is low. However, when the capacity reduces to the critical flow set up in the ramp metering controller, Ramp metering loses its efficiency. The parameters in the ramp metering controller therefore, require an update. For further research, it is recommended to simulate the same scenarios with different ramp metering algorithms. Since the functions of the algorithms are different, there might be other robust control algorithms for automated vehicles. Besides, another limitation of this thesis is that the automation system in level 2 vehicles is defined as Adaptive Cruise Control (ACC) plus Lane Change Assistance (LCA) system. Other partial automation systems may have a different effect on the performance of ramp metering. This thesis can be expanded by research the ramp metering performance under various types of partial automation systems.
Object detection is one of the most important research topics in autonomous vehicles. The detection systems of autonomous vehicles nowadays are mostly image-based ones which detect target objects in the images. Although image-based detectors can provide a rather accurate 2D position of the object in the image, it is necessary to get the accurate 3D position of the object for an autonomous vehicle since it operates in the real 3D world. The relative position of the objects will heavily influence the vehicle control strategy. This thesis work aims to find out a solution for the 3D object detection by combining the Lidar point cloud and camera images, considering that these are two of the most commonly used perception sensors of autonomous vehicles. Lidar performs much better than the camera in 3D object detection since it rebuilds the surface of the surroundings by the point cloud. What’s more, combing Lidar with the camera provides the system redundancy in case of a single sensor failure. Due to the development of Neural Network (NN), past researches achieved great success in detecting objects in the images. Similarly, by applying the deep learning algorithms to parsing the point cloud, the proposed 3D object detection system obtains a competitive result in the KITTI 3D object detection benchmark.
...
Object detection is one of the most important research topics in autonomous vehicles. The detection systems of autonomous vehicles nowadays are mostly image-based ones which detect target objects in the images. Although image-based detectors can provide a rather accurate 2D position of the object in the image, it is necessary to get the accurate 3D position of the object for an autonomous vehicle since it operates in the real 3D world. The relative position of the objects will heavily influence the vehicle control strategy. This thesis work aims to find out a solution for the 3D object detection by combining the Lidar point cloud and camera images, considering that these are two of the most commonly used perception sensors of autonomous vehicles. Lidar performs much better than the camera in 3D object detection since it rebuilds the surface of the surroundings by the point cloud. What’s more, combing Lidar with the camera provides the system redundancy in case of a single sensor failure. Due to the development of Neural Network (NN), past researches achieved great success in detecting objects in the images. Similarly, by applying the deep learning algorithms to parsing the point cloud, the proposed 3D object detection system obtains a competitive result in the KITTI 3D object detection benchmark.
LMI-based Stability Analysis for Learning Control
Deep Neural Networks and Locally Weighted Learning
Master thesis
(2018)
-
Konstantinos Kokkalis, Sebastian Trimpe, Jens Kober, Hans Hellendoorn, Alfredo Nunez Vicencio, Wei Pan
Learning capabilities are a key requisite for an autonomous agent operating in dynamically changing and complex environments, where pre-programming is not anymore possible. Furthermore, it is essential to guarantee that the learning agent will act safely by considering its stability properties. In this thesis, novel conditions are proposed, aiming to examine stability of the learned dynamics for two important model classes; namely Rectified Linear Unit (ReLU) Deep Neural Networks (DNNs) and Locally Weighted Learning (LWL). For the former method, a theoretical and computational framework is developed by establishing an equivalence between ReLU DNN models and Piecewise Affine (PWA) systems. This allows to leverage well-known tools of PWA system analysis, and consequently compute, characterize equilibria and determine their region of attraction for ReLU DNNs. Due to their increased complexity, a structured search for appropriate stability conditions was performed for LWL methods until the optimal trade-off between conservativeness and computational efficiency was obtained. These stability conditions are given as Linear Matrix Inequality (LMI) problems and they consist the first stability results in literature for these two model classes. Their efficacy is assessed in numerical and real-world dynamical systems and it is shown that the proposed LMIs are not unreasonably conservative, as they can evaluate accurately the stability properties of these two representations. Finally, this work demonstrates how to formulate appropriate stability conditions for learning methods in a principled manner.
...
...
Learning capabilities are a key requisite for an autonomous agent operating in dynamically changing and complex environments, where pre-programming is not anymore possible. Furthermore, it is essential to guarantee that the learning agent will act safely by considering its stability properties. In this thesis, novel conditions are proposed, aiming to examine stability of the learned dynamics for two important model classes; namely Rectified Linear Unit (ReLU) Deep Neural Networks (DNNs) and Locally Weighted Learning (LWL). For the former method, a theoretical and computational framework is developed by establishing an equivalence between ReLU DNN models and Piecewise Affine (PWA) systems. This allows to leverage well-known tools of PWA system analysis, and consequently compute, characterize equilibria and determine their region of attraction for ReLU DNNs. Due to their increased complexity, a structured search for appropriate stability conditions was performed for LWL methods until the optimal trade-off between conservativeness and computational efficiency was obtained. These stability conditions are given as Linear Matrix Inequality (LMI) problems and they consist the first stability results in literature for these two model classes. Their efficacy is assessed in numerical and real-world dynamical systems and it is shown that the proposed LMIs are not unreasonably conservative, as they can evaluate accurately the stability properties of these two representations. Finally, this work demonstrates how to formulate appropriate stability conditions for learning methods in a principled manner.
In recent years, enormous progress has been made in the field of automated driving. As a consequence, automated driving technologies are becoming increasingly popular. Research on comfort for autonomous vehicles, however, is still limited and unexplored. Some researchers address the comfort issue in path planning by velocity profiles, which regulates the instantaneous values of vehicle acceleration and jerk. Meanwhile, the actuator response to external disturbances and the inaccurate following can result in the violation to the pre-designed path, and therefore causes an uncomfortable driving experience. In order to tackle the passenger comfort issue from the perspective of path following control, this study proposes a frequency shaped model predictive control scheme that is (1) robust under external disturbances and (2) able to optimize passenger comfort by regulating the vehicle lateral acceleration with respect to its corresponding frequency. The frequency is selected based on the comfort evaluation criteria proposed in ISO 2631. Further, the proposed controller is tested in three simulation scenarios, compared to three baseline controllers with respect to tracking accuracy and driving comfort. Finally, our analysis shows that the FSMPC controller can improve driving comfort, especially at the velocity higher than 60 km/h.
...
In recent years, enormous progress has been made in the field of automated driving. As a consequence, automated driving technologies are becoming increasingly popular. Research on comfort for autonomous vehicles, however, is still limited and unexplored. Some researchers address the comfort issue in path planning by velocity profiles, which regulates the instantaneous values of vehicle acceleration and jerk. Meanwhile, the actuator response to external disturbances and the inaccurate following can result in the violation to the pre-designed path, and therefore causes an uncomfortable driving experience. In order to tackle the passenger comfort issue from the perspective of path following control, this study proposes a frequency shaped model predictive control scheme that is (1) robust under external disturbances and (2) able to optimize passenger comfort by regulating the vehicle lateral acceleration with respect to its corresponding frequency. The frequency is selected based on the comfort evaluation criteria proposed in ISO 2631. Further, the proposed controller is tested in three simulation scenarios, compared to three baseline controllers with respect to tracking accuracy and driving comfort. Finally, our analysis shows that the FSMPC controller can improve driving comfort, especially at the velocity higher than 60 km/h.