In the past few years, there has been much research in the field of Autonomous Vehicles (AV). If AVs are implemented in our daily lives, this could have many advantages. Before this can happen, safe driver models need to be designed which control the AVs. One technique that is suitable to create these models is Reinforcement Learning (RL). A problem here is that an RL agent usually needs to execute random actions during training, which is unsafe when driving an AV. Two shields are proposed to solve this problem: a Safety Checking Shield (SCS) and a Safe Initial Policy Shield (SIPS). The SCS checks whether an action is safe by predicting the future state after taking that action and checking whether that future state is safe. The SIPS checks whether an action is safe by comparing it to a safe action from a Safe Initial Policy. Based on the safety of the current state and this action, a safe range of actions is created in which the chosen action must fall. Furthermore, two shield-based learning techniques are proposed which are part of the RL algorithm and allow the agent to learn to avoid proposing actions that would be overruled by a shield. For the first method, experiences are fabricated, and for the second method, an alternative loss function is adopted. In the CARLA driving simulator, two scenarios were created to test the systems. In the first scenario, the agent needs to learn to drive straight, and in the second scenario, it needs to learn to not hit other vehicles on a straight road. The two shields are built around a Double Deep Q-Network (DDQN) and compared to it. It is shown that both shielding systems have zero collisions during training and execution, while having a similar or even better performance in terms of efficiency in comparison to the baseline DDQN. Furthermore, it is shown that both shield-based learning techniques effectively enable the agent to learn to not propose unsafe actions. ... Read more

One of the greatest challenges in marketing is measuring the return of investment of a marketing campaign and translating that into a strategy. Companies spend a lot of money on marketing without knowing how eective certain marketing campaigns are. To solve this problem for bunq, we will be using machine learning to create a marketing attribution system which outputs the optimal parameters for advertisements, based on data from all previous bunq advertisements. This tool can be used by the marketing department of bunq to increase its eciency. The marketing attribution project consists of three parts: the machine learning model itself, the input data of the machine learning model and the system through which people can get output of the model. The machine learning model is created by a data scientist at bunq. The model uses supervised learning, a method that uses a set of annotated training data as a supervisor for learning patterns. We specically make use of deep learning models that use regression to nd either the expected amount of clicks or the cost per acquisition of an advertisement. The results of these models are presented as a JSON le containing the best n advertisement options and their features. The input data to train the machine learning model was created by us. One component of the input data are the so-called touchpoints from Adjust. Adjust is an advertisement tracking company, which helps bunq with gathering data about all online encounters people had with bunq, like clicks on bunq advertisements or visits to the bunq website. The Adjust data gives the machine learning model information about how often an advertisement has been seen or clicked on, but it does not give information about how ecient an advertisement was in terms of the gained users. To solve this, we wrote an algorithm that anonymously matches the Adjust data to user data in the bunq database, based on IP-address and timestamps. The more links an advertisement has with users, the more ecient it is since it has been part of a process that convinced many users to become a bunq user. With this input data the machine learning model can be trained. The second part of the project is creating a connection to the machine learning model in such a way that the marketing department can use it. We created a python server that accepts calls from the bunq backend and sends the calls to the model, which is written in Java. It will then pass on the response of the model back to the bunq backend. In the python server, we use a bayesian technique to determine the best inputs for the marketing attribution machine learning model, to nally get the best possible parameters for a certain advertisement. All code in the backend is written in PHP and .json in a very clear Model View Controller structure, with strict bunq coding guidelines. Testing is done with PHPUnit tests.
... Read more