Intelligent manufacturing has become increasingly important in the food packaging industry due to the growing demand for enhanced productivity and flexibility while minimizing waste and lead times. This work explores the integration of such manufacturing in automated secondary robotic food packaging solutions that transfer food products into containers using pick-and-place robots. A major problem in these solutions is varying product supply caused by prior machinery. As a result productivity drops drastically when conveyor belt speeds are not optimally controlled. Conventional heuristic-based engineered approaches are used to address this issue but are inadequate, leading to noncompliance with industry's requirements. Reinforcement learning, on the other hand, has the potential of solving this problem by learning quick and predictive decision-making behavior based on experience. However, the lack of research in reinforcement learning for complex industrial robotic problems, limits its adoption in industry. Therefore, this work aims to investigate the feasibility of reinforcement learning in the robotic packaging industry. We propose a reinforcement learning framework, with policy inference in a highly complex control scheme, designed to optimize the conveyor belt speed of the secondary robotic packaging solution using real-world product supply data. The framework exceeds the 99.8 percent performance requirement and maintains quality at the required 100 percent when tested on real-world data. Compared to the current heuristic-based solution, our proposed framework improves productivity, has smoother control and reduces code execution time.

Interactive imitation learning refers to learning methods where a human teacher interacts with an agent during the learning process providing feedback to improve its behaviour. This type of learning may be preferable with respect to reinforcement learning techniques when dealing with real-world problems. This fact is especially true in the case of robotic applications where reinforcement learning may be unfeasible as there are long training times and reward functions can be hard to shape/compute.

The present thesis focuses on interactive learning with corrective feedback and, in particular, in the framework Deep Corrective Advice Communicated by Humans (D-COACH), which has successfully shown to be advantageous in terms of training time and data efficiency. D-COACH, a supervised learning method whose policy is represented by an artificial neural network, incorporates a replay buffer where samples of states and corresponding labels gathered by the agent's policy from human feedback are stored and replayed. However, this causes conflicts between the data in the buffer because samples collected by older versions of the policy may be contradictory and could deteriorate the performance of the current policy. In order to reduce this issue, the current implementation of D-COACH uses a first-in-first-out buffer with limited size, as the older the sample is, the more likely it is to deteriorate the performance of the learner. Nonetheless, this limitation propitiates catastrophic forgetting, an inherent tendency of neural networks to forget what they have already learnt, and that can be mitigated by replaying information gathered during all the stages of the problem. Therefore, D-COACH suffers from a trade-off between reducing conflicting data and avoiding catastrophic forgetting. The fact that D-COACH limits the size of its buffer automatically restricts the types of problems that it can solve, given that, if the problem is too complex (i.e. it requires large amounts of data), it simply will not be able to remember everything.

If we want to utilise a buffer to train data intensive tasks with corrective feedback, a new method is needed to solve the problem of using information gathered by older versions of the policy. We propose an improved version of D-COACH, which we call Batch Deep COACH (BD-COACH, pronounced “be the coach”). BD-COACH incorporates a human model module that learns the feedback from the teacher and that can be employed to make corrections gathered by older versions of the policy still useful for batch updating the current version of the policy.

To compare the performance of BD-COACH with respect to D-COACH, three simulated experiments were done using the open-source Meta-World benchmark, which is based on MuJoCo and OpenAI gym. Moreover, to validate the proposed method in a real setup, two planar manipulation tasks were solved using a seven degrees of freedom KUKA robot arm.

Furthermore, we present an analysis between on-policy and off-policy methods both in the fields of reinforcement learning and in imitation learning. We believe there is an interesting simile between this classification and the problem of correctly implementing a replay buffer when learning from corrective feedback.

Learning from demonstration is a technique where the robot learns directly from humans. It can be beneficial to learn from humans directly because humans can easily demonstrate complex behaviors without being experts in demonstrating required tasks. However, it can be challenging to gather large amounts of data from humans because humans often get tired, bored, or lose focus while giving many demonstrations. Therefore, the learning algorithms require to be as data-efficient as possible. To solve sequential tasks using few human demonstrations, the Transporter Network was introduced.

The Transporter Network consists of two networks, namely the pick network and the place network. Both these networks have a fully convolutional architecture which has the property of being translationally equivariant. Translational equivariance is well suited for estimating pick locations because the predicted pick location changes corresponding to the change in the object's position in the image. However, the performance of the transporter network on sequential tasks depends on the receptive field of the fully convolutional network because each pixel in the output should depend on each pixel in the input image. This correlation is important for the network to make predictions based on the overall configuration of objects in the input image. A larger receptive field would give a prediction by correlating large amount of pixels in the input image, however increasing the receptive field could result in loss of resolution which in turn leads to information loss in the latent space. Therefore, it is essential to select an appropriate size for the receptive field such that there is minimal loss of resolution in the latent space of the fully convolutional network. This limitation of selecting the appropriate size of the receptive field was also observed in the preliminary experiments on the pick network of the Transporter Network.

In this work, a SEquential Attention Network (SEA Net) is introduced to overcome the dependence of the Transporter Network on the receptive field size for solving sequential task. SEA Net is a variation of the pick network of the Transporter Network. SEA Net works under the assumption that the set of objects to be picked are known beforehand. Based on this assumption, the SEA Net learns to predict two things: 1) The pick location of all the objects. 2) "what" object to pick as a function of the current configuration of the objects in the environment. SEA Nets can be used independently to predict pick location sequentially and also be used as a replacement of the pick network in the Transporter Network. SEA Net is evaluated on two datasets, namely synthetic dataset and simulated robot dataset. The synthetic dataset has a top-down view of simple shape blocks. In contrast, simulated robot datasets are extracted using a pre-trained policy on the Mujoco simulators. The Object Keypoint Similarity (OKS) metrics is used to score the distance between the predicted pick point and the ground truth. The OKS Metric uses a standard deviation threshold (σ(threshold)). A strict standard deviation threshold (σ(threshold)) of 2 pixels and a lenient threshold (σ(threshold)) of 20 pixels is used because a strict threshold will evaluate the resolution of the prediction. In contrast, the lenient threshold will assess if the predicted pick point is, at least on the desired object. The model achieved an overall accuracy of 64% for a strict standard deviation threshold (σ(threshold)) of 2 pixels and accuracy of 85% for a lenient standard deviation threshold (σ(threshold)) of 20 pixels on the OKS metrics, on the synthetic dataset. The model could perform on a simulated robot dataset with an accuracy of 82% on a strict standard deviation threshold (σ(threshold)) of 2 pixels.

Interactive machine learning describes a collection of methodologies in which a human user actively participates in a novice agent’s learning process, through providing corrective or evaluate feedback or demonstrative actions. A primary assumption in these methods is that user input is at worst nearoptimal, however a realistic set of demonstrations will often contain conflicting or poor examples, which degrade the quality of the learnt policies. This project explores methods for the detection of such undesirable features in data and develops an algorithm for policy training with suboptimal demonstrations, while leveraging the generalisation and scalability qualities of artificial neural networks. Uncertainty estimation, which presents a structured approach for the quantification of a network’s confidence in the accuracy of its output, based on the observed training data, is applied for the detection of unwanted features in a demonstration dataset. The particular focus of this project is conflicting data resulting from scenarios with equivalent action choices, such as the obstacle avoidance setting. Following thorough testing on various environments, it is shown that novice policies may be trained to achieve a desired goal in multi-dimensional spaces with either discrete or continuous data, despite the presence of conflicts in this training data.