Apache Spark is a popular batch processing framework that is integrated into the ASML data analytics platform. However, having multiple users share the same resources creates fairness and efficiency problems in the scheduler, where some users may receive more resources than others, or the scheduler slows down all jobs equally. Apache Spark provides a built-in fair scheduler that attempts to mitigate these issues, but cannot account for changing user environments and has been shown to slow down overall job execution. Previous literature has already proposed improvements to the Spark fair scheduler, however, it does not account for user contexts to ensure fair resource distribution among users, and does not account for task skews that are present in the Spark ecosystem.


In this work, we present User Weighted Fair Queuing (UWFQ) scheduler that can minimize the response time of jobs in Apache Spark, while ensuring that users are allocated a fair share of resources that they equally distribute among their jobs. This is done by simulating a virtual fair scheduler, and assigning the highest priority to jobs that complete the earliest in the virtual scheduler. The algorithm is an extension to Cluster Fair Queuing designed by C. Chen et al., with an added fairness layer to ensure that each user is entitled to a fair share of resources. To address the problem of task skew present in Spark, we introduce runtime partitioning that can more effectively avoid task skew by splitting them into more partitions. We implement our scheduler in the Apache Spark framework and show that UWFQ with runtime partitioning can decrease the average response time of small jobs by up to 70% in certain workloads, while still providing fair resource allocation in critical scenarios.

This study has investigated the effect of the urban environment on pedestrian route choice behaviour, for walking routes to and from train stations in the Netherlands. To the best knowledge of the author, this is the first pedestrian route choice study which is performed at country-level scope. This was made possible by applying an innovative quasi-revealed preference approach. Respondents self-reported their route preferences through an online GIS survey. Outputs were subsequently analysed by overlaying route data with a range of open geodata sources and searching for statistical similarities between certain urban features and (in)frequently taken routes. Results from statistical testing are verified by testing these against a qualitative text-coding analysis. The study finds that pedestrians in the Netherlands have a strong preference for the route which they perceive to be fastest, especially when they are en-route to a train station. Pedestrians dislike features which they perceive as ’slowing’ them down such as traffic lights. Detours are found somewhat more often on from-station routes, with such detours leading via environments which are attractive for social reseasons such as shops or busy streets. The method was found to work fairly well and should now be tested on a larger population sample with more variables.