Previous research has in reinforcement learning for traffic control has used various state abstractions. Some use feature vectors while others use matrices of car positions. This paper first compares a simple feature vector consisting of only queue sizes per incoming lane to a matrix of car positions. Then it investigates if knowledge can be transferred from a simple agent using the feature vector abstraction to a more complex agent that uses the position matrix abstraction.We find that training cannot be sped up by first training an agent with the feature vector abstraction and then reusing this Q-function to train an agent with the position matrix abstraction. The simple agent does not take considerably fewer samples to converge, and the total time needed to first train the simple agent and then transfer exceeds the time needed to train the complex agent from scratch.

By increasing the step frequency of the runners, it is possible to reduce the risk of injuries due to overload. Techniques like auditory pacing help the athletes to have better control over their step frequency. Nevertheless, synchronizing to a continuous external rhythm costs energy. For this reason, the use of intermittent pacing may be more energy-efficient and more user-friendly for the athlete. We propose using experimental data from previous studies, that analyzed the response of runners to intermittent pacing, to find the most efficient approach for providing the pacing. For this purpose we use reinforcement learning techniques to learn and train our target behavior. This behavior is represented as the target policy and the experimental data is assumed to be sampled using a stochastic sampling policy. However, using only a single batch of initial training data presents a problem due to the continuously increasing difference between the initial sampling policy and the target policy being learned. The use of a batch off-policy algorithm with a standard deviation correction (OPPOSD) presented in (Liu et al., 2019) is then proposed. This algorithm benefits from the advantages of the sampling efficiency characteristic of the off-policy approaches and also introduces a fixing term to tackle the mismatch between the policies. To train and evaluate the learned policies based on the algorithm, a pace behavior simulator was developed from the data of the experiments. A Markov Decision Problem (MDP) was defined on top of the simulator that determines the rules of the pacing environment that the algorithm is set to learn. After translating the experimental data into MDP-like transitions, the OPPOSD algorithm is able to learn a relatively good target policy for the pacing problem. For a future application, the resulting trained model could be deployed for real runners while still having a continuous improvement of the policy in an on-policy or off-policy approach.

In order to do something useful, you should know what to do and how to do it. The same goes for robots or other machines, which are also referred to as agents. Advances in e.g. technology and science have made such agents more and more sophisticated and capable. This opens up a plethora of possibilities for solving problems or automating processes. However, as agents and the problems they solve become more and more sophisticated, the software that controls these agents naturally also becomes more complex. Especially when considering systems of multiple cooperating agents. With rapidly increasing amounts of agents, observations and capabilities to take into account, the plans used to come to decisions become increasingly infeasible to create by hand. Thus automated methods are necessary to create such plans. The approach this thesis uses is inspired by the ability of human intelligence to abstract away pedantic details when planning. We do not for example include which muscles we will utilize, when we make a plan for picking up groceries. Deciding to get groceries is on a completely different level than deciding how many centimeters we want to move a limb in a particular direction. Abstracting away such details greatly reduces the size of any potential plan. Which makes plans easier to construct, while the scope of the problem remains the same. Thus, utilizing this notion of different levels of abstraction during planning, is the approach this thesis uses to improve planning for agents. This is done by extending the Macro Decentralized Partially Observable Markov Decision Process (MacDecPOMDP) framework to allow for an arbitrary number of levels of abstraction.