For traditional, manual real estate appraisals, the appraiser is required to provide a number of comparable properties (the 'Comps'). These comps act as a benchmark for the valuation as well as a provider of context in the final appraisal report. Traditionally, these comps are selected manually by an appraiser based on recent transactions within a ten mile radius. This manual selection is biased by the appraiser's market knowledge and the amount of transactions in the area. To replace this process, we developed an automated comparable selection service that does so based on objective characteristics, without restricting itself to a small spatial and/or temporal slice of the market. Comparable selection does not have an objective ground truth, which complicates or even prohibits the use of many machine learning algorithms that could otherwise have been used. Additionally, the outcome of the service needs to be explainable to its users — it cannot be completely opaque. Finally, our service needed to integrate with a streaming data platform, with incremental new data that arrives continuously and needs to be incorporated into the service's model and output. Our research phase focused on three aspects: determining the possible algorithms for selecting comparable properties given the constraints of explainability and a streaming environment, how to explain the output of the chosen algorithm to the user, and how to build a service around the chosen model that consumes a stream of input data and can generate a set of comparable properties ad-hoc. Our process was based on agile methodology, with two-week sprints in which we gradually expanded our service into a fully functional proof of concept. Challenges were encountered while developing the logic to incrementally construct a database of real estate properties from the stream of data. These were resolved by switching from a document store to an RDF database, which better matched the flow of data coming in. The final product consists of several microservices, each of which handles part of the problem domain and can be scaled out independently. A REST API and a web front-end are accessible to its users. The system was tested using both unit tests and end-to-end testing, whereas the model was refined by scoring output on closeness of features indicative of similarity. As a future improvement, the current model used by the service is fairly simple and can most likely be improved upon once more data is available. Additionally, due to the lack of a ground truth it will be important to tune both comparable selection and explanation metrics in response to user testing.

There is a correlation between music and affect which researchers try to define and use in technology to improve healthcare and users' experience in music-related technology. However, since affect is a complex term there is not a specified way on how to represent different affective states in systems. A systematic literature review was performed to give an overview of affect representation schemes (ARS) used in music affect prediction. All the studies included in this survey were found in Scopus, Web of Science and IEEE Explorer and focus on automatic affect prediction in music. What is more, the literature written in a different language than English and not from the Computer Science field was excluded. Considering the time constraint of 10 weeks, an additional feasibility filtering was conducted. That is only the documents which mention common benchmark datasets for this task were included. The datasets were chosen after the research of related work and different literature reviews related to this topic. We were aware that it might include a bias correlated to dataset popularity. At the final stage of filtering, 113 records were chosen for data extraction from which 51 were used in the analysis due to time constraints and a single person conducting this research. After synthesising and analysing the extracted data we observed that the schemes can be distinguished between dimensional and categorical approaches and both are similarly popular. A rare but existing ARS are developed by combining both scheme types in a unique representation. What is more, there is no easily visible trend over time in the usage of certain schemes and only 66% of the reviewed studies support their choices with psychological theories. While conducting this research, we encountered limitations in terms of time and the research group, so we leave a gap for future improvements in this project.

In human-human interactions, the majority of information is conveyed through body language, specifically facial expressions. Consequently, researchers have been interested in improving human-computer interactions through developing systems with automatic understanding of body language and facial expressions.  This technology is especially useful due to its broad range of applications in fields such as healthcare, education, and safety & security. Vision-based automatic affect recognition (AAR) systems aim to predict a subject’s affective state based on visual input such as image or video. These systems analyze and classify subjects’ facial expressions and body language using affect representation schemes (ARS), most often classified as either categorical or dimensional. This paper explores the current state of ARS used in vision-based AAR through a systematic literature review following PRISMA guidelines. We selected 53 papers from WebOfScience according to our eligibility criteria which included computer science papers written in English proposing a vision-based AAR system targeting single subjects, and excluded studies dealing exclusively with micro-expressions or group affect recognition. Additionally, given the time limitation imposed on this research we excluded papers that were not readily accessible with our TU Delft license, used multimodal input, or did not use a dataset included in our  predefined list. For this exploration we specifically look at the schemes used, the popularity and trends of usage, motivations, and psychological basis. From the 53 reviewed papers, all of the papers target utilitarian emotions using at least one discrete ARS. The most commonly used schemes classify affective states into happiness, sadness, fear, anger, surprise, and disgust. While the majority of papers are lacking in providing explicit reasoning for their choices, most ARS are based grounded in psychological theories. Our results show an established norm within this area of research. However, they also evidence a lack of displayed critical thought in the selection of schemes. This oversight limits potential for future AAR research.

Physiological signals, such as Electroencephalogram (EEG), Glavic Skin Response (GSR), or Body Temperature, are common inputs for Automatic Affect Recognition (AAR) systems. One of the crucial elements of AAR is the Affect Representation Scheme (ARS) used to define the affective states recognized by the system (e.g., happiness, sadness, fear, anger). Throughout the years many AAR reviews have been published. However, most of them do not include a detailed analysis of ARSs and the motivation behind them. This paper aims to fill this knowledge gap by performing a Systematic Literature Review (SLR) of Computer Science papers that propose a Physiological-signal-based AAR (PAAR) system. We explore how researchers discuss and choose an ARS and whether they base it on actual psychological theories. Eligible papers are retrieved from 4 databases: Web Of Science, IEEExplore, Scopus, and ACM Digital Library. Due to time limitations, the review is done rapidly and some additional search constraints are applied for feasibility. The most significant restrictions are: considering papers published since 2020 and performing experiments on specific benchmarking datasets. We take these constraints and their possible impact into consideration when interpreting the results. The presented review procedure can be stripped from the additional filters and reused for a full review. In total 115 papers are processed. The majority of papers introduce an EEG-based emotion recognition system. The analysis of the extracted information reveals that dimensional ARSs, in particular, Valence/Arousal model is the most popular. Moreover, authors often choose to reduce the dimensions to high/low values. Categorical ARSs are less frequent and usually are adopted from the dataset. Lastly, the authors do not provide extensive motivation for the choice of ARS and rarely refer to psychological theories.

In practice, many applications like traffic monitoring and smart grids rely on computing functions on privacy-sensitive data. In order to protect privacy-sensitive data and still keep the ability to compute any arbitrary function, multi-functional privacy-preserving data aggregation schemes have been created. These schemes, however, can be abused by malicious users, leading to incorrect results. Existing literature mostly provides aggregation schemes which are either multi-functional or support malicious user detection, but to the best of our knowledge, there is only one scheme that provides both. This scheme requires each user to send a number of ciphertexts linear in the size of the aggregation function's domain. Furthermore, that scheme is not collusion-resistant. In this thesis, we design a multi-functional privacy-preserving data aggregation scheme with malicious user detection. In contrast to existing schemes in literature, the amount of messages is independent of the size of the aggregation function's domain, it does not rely on a trusted authority and it is collusion-resistant as long as at least two users are honest-but-curious.

In order to stay undetected and keep their operations alive, cyber criminals are continuously evolving their methods to stay ahead of current best defense practices. Over the past decade, botnets have developed from using statically hardcoded IP addresses and domain names to randomly-generated ones, so-called domain generation algorithms (DGA). Malicious software coordinated via DGAs leaves however a distinctive signature in network traces of high entropy domain names, and a variety of algorithms have been introduced to detect certain aspects about currently used DGAs. Today's detection mechanisms are evaluated for botnets that make the next obvious evolutionary step, and replace domain names generated from random letters with randomly selected, but actual dictionary words. It can be seen that the performance of state-of-the-art solutions that rely on linguistic feature detection would significantly decline after this transition, and an alternative novel approach to detect DGAs without making any assumptions on the internal structure and generating patterns of these algorithms is proposed.

Synthesizing audio-reactive videos to accompany music is challenging multi-domain task that requires both a visual synthesis skill-set and an understanding of musical information extraction. In recent years a new flexible class of visual synthesis methods has gained popularity: generative adversarial networks. These deep neural networks can be trained to reproduce arbitrary images based on a dataset of about 10000 examples. After training, they can be harnessed to synthesize audio-reactive videos by constructing sequences of inputs based on musical information. Current approaches suffer from a few problems which hamper the quality and usability of GAN-based audio-reactive video synthesis. Some approaches consider only a small number of possible musical inputs and ways of mapping these the GAN's parameters. This leads to weak audio-reactivity which has a similar motion characteristic across all musical inputs. Other approaches do harness the full design space, but are difficult to configure correctly for effective results. This thesis aims to address the tradeoff between audio-reactive flexibility and ease of attaining effective results. We introduce multiple algorithms that explore the design space by using machine learning to generate sequences of inputs for the GAN. To develop these machine learning algorithms, we first introduce a metric, the audiovisual correlation, that measures the audio-reactivity in a video. We use this metric to train models based only on a dataset of audio examples, avoiding the need of a large dataset of example audio-reactive videos. This self-supervised approach can even be extended to optimize a single audio-reactive video directly, removing the need to even train a model beforehand. Our evaluation of the methods shows that our algorithms out-perform prior work in terms of their audio-reactivity. Our solutions explore a wider range of the audio-reactive space and do so without the need for manual feature extraction or configuration.

In the machine learning research community, significant importance is given to the optimization of techniques which are employed once a benchmark dataset is given. However, less importance is assigned to the quality of these datasets and to how these datasets are obtained. In this work, we look into annotation practices in the research area of affective computing, analysing datasets of emotion classification tasks from text, video, audio, EEG data and more. We find annotation practices of varying quality and recommend that annotation practices be improved, especially with regard to multiple annotator overlap.

This research examines transparency between ICASSP conference papers and the dataset documentations related to the datasets' annotation practices. Top-cited 5 papers and 51 unique resources in total were considered. All of the selected papers utilized at least one dataset. For every paper, an extensive metadata search has done to reach the initial datasource of those datasets. These searches happened both within the paper contents such as sections and references along with outside the paper contents through the way of extensive web queries. Analysis of the papers published from 2021 and 2022 and their relevant datasets revealed varying levels of transparency. Original dataset creators provide comprehensive information, while papers using modified datasets offer limited details on initial annotations. Emphasizing the need for accountability, this study suggests that papers utilizing datasets should trace back to the initial dataset and provide explicit comments. The findings underscore the importance of ensuring sufficient information in initial datasets and promoting transparency and traceability in dataset annotation practices within the ICASSP community.

This systematic review investigates the practices and implications of human annotations in machine learning (ML) research. Analyzing a selection of 100 papers from the IEEE Access Journal, the study explores the data collection and reporting methods employed. The findings reveal a prevalent lack of standardization and formalization in the annotation process. Key details such as annotation sources, number of annotators, and formal instructions are frequently neglected, possibly compromising the quality and effectiveness of ML algorithms. Domain-specific implications are discussed, highlighting the need for comprehensive annotation practices in areas like medical diagnostics, language processing, and intelligent vehicle systems. The study contributes to the field by emphasizing the importance of standardized procedures and transparency in ML research. Future research is recommended to develop systematic annotation methodologies and examine the impact of subpar annotation on data quality.

As of this moment there is a lack of data about rainfall in cities. To collect such data, IBM has started the Sense Umbrella Connection and Desensitisation project. For this project an umbrella was equipped with a piezoelectric sensor and a Bluetooth device to record the rain that falls on the surface of this umbrella. One downside of this umbrella is that, besides rain, it will also record other sounds which include recognisable human speech. Because IBM values privacy, one of the tasks was to make sure that no recording containing recognisable human speech would be uploaded to a server. This bachelor project focuses on creating a mobile application to connect to the umbrella via Bluetooth and save the audio recordings of rain together with GPS data. After saving this data it would originally be analysed for presence of rain and it was also required to remove all human speech before it was sent from the phone to the server. Over the course of the project it became clear that because of a scarce set of sample data and the limited availability of audio processing libraries on Android, it would be difficult to process audio on Android devices. This is why the decision was made to upload all data, and the processing and analysing was moved to an external device supporting Java. The raw data will now be uploaded to a server which will make a database entry which includes GPS data, and saves the audio file. In the end an app has been developed that is able to gather data from the umbrella and send it to an external server. Additionally an implementation has been made to analyse the audio data gathered to classify which parts may contain rain. This project focused on developing an Android app and not on other operating systems due to time constraints.

Machine Learning models are nowadays infused into all aspects of our lives. Perhaps one of its most common applications regards recommender systems, as they facilitate users' decision-making processes in various scenarios (e.g., e-commerce, social media, news, online learning, etc.). Training performed on large volumes of data is what ultimately drives such a system to provide meaningful recommendations, and yet there has been observed a lack of standardized practices when it comes to data collection and annotation methods for Machine Learning datasets. This research paper systematically identifies and synthesizes such processes by examining existing literature on recommender systems. The review includes 100 most-cited papers from the most impactful venues within the Computing and Information Technology field. Multiple facets of the employed techniques are touched upon, such as reported human annotations and annotator diversity, label quality, and the public availability of training datasets. Recurrent use of just a few benchmark datasets, poor documentation practices, and reproducibility issues in experiments are some of the most striking findings uncovered by this study. A discussion is centered around the necessity of transitioning from reliance solely on algorithmic performance metrics in favor of prioritizing data quality and fit. Finally, valid concerns are raised when it comes to biases and socio-psychological factors inherent in the datasets, and further exploration of embedding these early in the design of ML models is suggested.

Using Recommender Systems with Evolutionary Algorithms is an extremely niche domain. It holds the key to enabling new user interaction designs, where users can effectively configure their experience with a Recommender System. This thesis answers important questions about the scientific aspects of its application to large-scale data through a rigorous experimental design. We use one of the largest publicly accessible music listening histories dataset to analyse if the methodology works well for large real-world tasks. The dataset has been used to simulate various real-world scenarios for the experimental design. The methodology fuses the recommendations generated by an unspecified number of recommenders. In this study, we have used three recommenders, which have specialised goals in terms of user-centric metrics. We use three different Evolutionary Algorithms to analyse the capability of different EA strategies for generating a near-optimal set of trade-offs on user-centric metrics. We have performed elaborate qualitative and quantitative analyses of the system to understand how various aspects of the system affect the final set of solutions.

Affective Video Content Analysis aims to automatically analyze the intensity and type of affect (emotion or feeling) that are contained in a video and are expected to arise in users while watching that video. This study aims to provide a systematic overview of various affect representation schemes utilised by researchers in the field of Affective Video Content Analysis and the reasons behind their choice. The main objectives of the study were to investigate the diversity of affect representation scheme types, their popularity over time, the basis of their selection, and the relationship between input data sources, in terms of direct and implicit analysis, and scheme types. Following the PRISMA guidelines to conduct a systematic literature review, a total of 45 papers were included in the study which were original journals and conference proceedings in the field of Affective Video Content Analysis and that were related only to Affective Movie Content Analysis published in English after 2008. Papers concerning Video Emotion Recognition were excluded from the review. The findings reveal that dimensional, categorical, and combined approaches are commonly used in this field, with the dimensional approach based on valence and arousal being the most prevalent. However, there is no significant trend in the popularity of affect representation scheme types over time. The study highlights the lack of clear motivation and explicit justifications for scheme selection, emphasizing the need for transparency and the inclusion of psychological theories as a basis for scheme choices. Additionally, the study found that audiovisual data was the most commonly used input compared to physiological signals and visible behavioural data. The study acknowledges its limitations, including time constraints and single researcher involvement, and suggests allocating more time and involving multidisciplinary teams for comprehensive insights.

Auto-tagging systems can enrich music audio by providing contextual information in the form of tag predictions. Such context is valuable to solve problems within the MIR field. The majority of re- cent auto-tagging research, however, only considers a fraction of tags from the full set of available annotations in the original datasets. Because of this restriction, potential relationships between tags remain unconsidered and tagging may be less rich. These relationships suggest alternative ways to establish an auto-tagging system. For instance, a few accurate annotations from experts can improve the richness and quality of the auto-tagging system by providing explicit context in addition to audio content features. In this work, we propose an adaptation to the auto-tagging task, semi auto-tagging, to demonstrate such potential. In our framework, tags are allowed as contextual input to the tag pre- diction system in addition to audio content information. The system then suggests additional relevant tags. We implement two models that fit within the framework: content-aware matrix factorization and graph convolutional networks. To see whether we can improve upon a traditional auto-tagger, we compare these models with a multilayer perceptron as a baseline. Experimental results show that semi auto-tagger models can predict relevant tags both in the absence and presence of an audio content feature, and can predict tags for previously unseen songs similarly to an audio content auto-tagger. Based on a tag embedding comparison, we find that semi auto-tagger models can better learn implicit relationships between tags with a similar text string representation when compared to the baseline.

Occlusion degrades the performance of human pose estimation. In this paper, we introduce targeted keypoint and body part occlusion attacks. The effects of the attacks are systematically analyzed on the best-performing methods. In addition, we propose occlusion specific data augmentation techniques against keypoint and part attacks. Our extensive experiments show that human pose estimation methods are not robust to occlusion and data augmentation does not solve the occlusion problems.

Mathematical fairness notions introduced in literature aim to make algorithmic decisions fair. However, their usage has been criticized in domains such as recidivism and lending for producing unfair decisions. Questions regarding fairness, which also have an important role in hiring are giving way to concerns about the increasing adoption of algorithmic decision-making systems in the field. However, there are no concrete studies linking mathematical notions to actual perceptions of fairness by people active in hiring and applicant selection. We aim to explore the understanding and alignment of existing fairness notions by organizational representatives in the context of early candidate selection in hiring. Towards that, we interviewed 17 professionals from executive functions, talent acquisition, HR, I/O psychology, and diversity and inclusion operations in The Netherlands. By designing user-friendly illustrations and explanations in the context of early candidate selection in hiring, we explore their ratings and responses to six fairness notions on understandability, perception of fairness, perception of diversity, and applicability. Our qualitative investigation suggests that these fairness notions raise three concerns. One, they lack additional contexts such as a company's size or diversity goals. Two, they give rise to several ethical and practical concerns such as lack of trust in the data, disadvantaging minorities, or the selection of unqualified applicants. Lastly, they act only as a small step towards fairness in the large hiring pipeline. We conclude that a qualitative approach in collaboration between designers, practitioners, and policymakers is the key to refinement and contextualization of future technologically enabled fair hiring policies. Our participants' intrinsic motivation to engage with the topic of fairness strengthens our case.

Police officers are exposed to many potentially traumatising and stressful situations, but they do not always find the right mental health help they need. In this thesis, conversational agent Robyn is developed to help officers keep an eye on their mental well-being and find help when needed. An evaluation approach is also presented. The conversational agent applies the Stress First Aid method to determine stress indicators together with the officer. The design of the agent followed the guidelines for Stress First Aid. The agent uses text input and output and went through an iterative design process using expert feedback. The experts provided feedback on both text and structure of the conversation. Based on their input, informal language was chosen for the presentation of the agent. The lessons for similar projects drawn from this design process are managing expectations from users, scoping broad questions, getting feedback from users on wording and not to push people when discussing suicide. The proposed evaluation would determine if the agent is effective at improving people's awareness of their mental well-being status. This is done by comparing perceived mental well-being before and after interaction to reported mental well-being. To achieve this, a survey to measure perceived mental well-being was developed. The evaluation also provides insight in what aspects of the agent need the most improvement in future iterations.

Comparing spreadsheet files is a new, unexplored research domain in computer science. Methods for regular file comparison are not straightforwardly applicable to spreadsheet files, because they are fundamentally different. Spreadsheets are binary files, the structure of spreadsheets is two-dimensional, they contain both data and calculations, and the content exists on different abstraction levels. Fundamental challenges in the spreadsheet comparison problem include: change propagation, performance, 2D alignment, the grouping of data, and movement detection. One simple modification can change the whole structure and model of the spreadsheet. The aim of a good file comparison method is to show the actual changes made by an end-user, not the propagated changes. In this thesis, a new pipeline-based approach for comparing two spreadsheet files is proposed. The spreadsheet comparison is solved in three phases: (1) structure analysis, (2) change detection, and (3) change aggregation. The essential element in this approach is passing on information from the structure analysis to the change detection. In addition, the detected changes are forwarded to the aggregation such that changes are converted into clear, understandable differences. The final comparison result, therefore, provides information at different levels of abstraction. New solutions like cell hashing, an optimized 2D alignment using longest common subsequences, and different algorithms for comparing worksheets, defined names, rows, columns and cells have resulted in a state-of-the-art spreadsheet comparison approach. In addition, this thesis presents a prototype demonstrating the proposed concepts in practice. This tool, called CompareXL, is a stand-alone application that can compare two spreadsheet files. The user interface is built with a multi-level interaction design, showing the comparison results intuitively to the end-user. Spreadsheets are commonly used and many problems arise from spreadsheet versioning issues. Spreadsheet users have user needs related to overview, validation, completeness, error resolving, visualization, and evolution. A spreadsheet comparison program is helpful to address these user needs. This research shows that it is beneficial to involve users in the exploration of solutions, because comparing spreadsheets is not only a technical problem but also a user problem. The outcome of this research offers multiple directions for future work, with the ultimate goal that all spreadsheet version problems will soon belong to the past.

High-level music classification tasks such as automatic music mood annotation impose several challenges, both from a psychological and a machine learning point of view. Ground truth labels for these tasks at hand are hard to define due to the abstract and aesthetic nature of the data, being largely dependent on human psychology and perception. Such labels, however, are required in training and validation sets when traditional machine learning methods are used. Furthermore, due to copyright restrictions which prevent the sharing of commercial music audio, such classifiers have to work with pre-computed music audio features which are known to be somewhat unstable. Due to the challenges inherent to high-level music classification, the following questions arise: do high-level music classifiers actually perform as well as we believe? And can we trust their output to be a solid foundation for future research? This work analyzes the performance of high-level music classifiers using metrics based on label stability, label agreement and distributional differences, all of which do not depend on any problematic ground truth labels, on a dataset which combines data from AcousticBrainz and Spotify. Unexpected patterns in classifier outputs are uncovered, indicating that these outputs should not be taken as absolute truth and do not form a solid foundation for further research. The improvement of these high-level music classifiers is a multidisciplinary effort for which better evaluation methods are required. To this end, several approaches for more comprehensive classifier testing are presented, based on best practices in psychology and software testing. These approaches are not constrained to the field of Music Information Retrieval and can be applied to evaluate classifiers in other domains as well.