Apache Spark is a popular batch processing framework that is integrated into the ASML data analytics platform. However, having multiple users share the same resources creates fairness and efficiency problems in the scheduler, where some users may receive more resources than others, or the scheduler slows down all jobs equally. Apache Spark provides a built-in fair scheduler that attempts to mitigate these issues, but cannot account for changing user environments and has been shown to slow down overall job execution. Previous literature has already proposed improvements to the Spark fair scheduler, however, it does not account for user contexts to ensure fair resource distribution among users, and does not account for task skews that are present in the Spark ecosystem.


In this work, we present User Weighted Fair Queuing (UWFQ) scheduler that can minimize the response time of jobs in Apache Spark, while ensuring that users are allocated a fair share of resources that they equally distribute among their jobs. This is done by simulating a virtual fair scheduler, and assigning the highest priority to jobs that complete the earliest in the virtual scheduler. The algorithm is an extension to Cluster Fair Queuing designed by C. Chen et al., with an added fairness layer to ensure that each user is entitled to a fair share of resources. To address the problem of task skew present in Spark, we introduce runtime partitioning that can more effectively avoid task skew by splitting them into more partitions. We implement our scheduler in the Apache Spark framework and show that UWFQ with runtime partitioning can decrease the average response time of small jobs by up to 70% in certain workloads, while still providing fair resource allocation in critical scenarios.

Indoor localization is an important field of research for advancing robotics and providing more accurate estimations of indoor locations for users. There are many indoor localization algorithm implementations, but many of them underperform under certain environmental changes or restrictions. This research will present a way of combining already existing indoor localization techniques to more accurately deduce the user’s location within a building. An experiment was conducted within campus building Pulse, where multiple fingerprints of locations where gathered, and then used to train and test the combined classification models. By fusing active acoustic location sensing and WiFi localization using weighted averaging, ensemble stacking, and 2-step localization, the combination of classifiers was able to outperform individual classifiers by up to 5% of localization accuracy. Additionally, 2-step localization and weighted averaging methods did not add any performance overhead.