In the machine learning research community, significant importance is given to the optimization of techniques which are employed once a benchmark dataset is given. However, less importance is assigned to the quality of these datasets and to how these datasets are obtained. In this work, we look into annotation practices in the research area of affective computing, analysing datasets of emotion classification tasks from text, video, audio, EEG data and more. We find annotation practices of varying quality and recommend that annotation practices be improved, especially with regard to multiple annotator overlap.

This research examines transparency between ICASSP conference papers and the dataset documentations related to the datasets' annotation practices. Top-cited 5 papers and 51 unique resources in total were considered. All of the selected papers utilized at least one dataset. For every paper, an extensive metadata search has done to reach the initial datasource of those datasets. These searches happened both within the paper contents such as sections and references along with outside the paper contents through the way of extensive web queries. Analysis of the papers published from 2021 and 2022 and their relevant datasets revealed varying levels of transparency. Original dataset creators provide comprehensive information, while papers using modified datasets offer limited details on initial annotations. Emphasizing the need for accountability, this study suggests that papers utilizing datasets should trace back to the initial dataset and provide explicit comments. The findings underscore the importance of ensuring sufficient information in initial datasets and promoting transparency and traceability in dataset annotation practices within the ICASSP community.

This systematic review investigates the practices and implications of human annotations in machine learning (ML) research. Analyzing a selection of 100 papers from the IEEE Access Journal, the study explores the data collection and reporting methods employed. The findings reveal a prevalent lack of standardization and formalization in the annotation process. Key details such as annotation sources, number of annotators, and formal instructions are frequently neglected, possibly compromising the quality and effectiveness of ML algorithms. Domain-specific implications are discussed, highlighting the need for comprehensive annotation practices in areas like medical diagnostics, language processing, and intelligent vehicle systems. The study contributes to the field by emphasizing the importance of standardized procedures and transparency in ML research. Future research is recommended to develop systematic annotation methodologies and examine the impact of subpar annotation on data quality.

Machine Learning models are nowadays infused into all aspects of our lives. Perhaps one of its most common applications regards recommender systems, as they facilitate users' decision-making processes in various scenarios (e.g., e-commerce, social media, news, online learning, etc.). Training performed on large volumes of data is what ultimately drives such a system to provide meaningful recommendations, and yet there has been observed a lack of standardized practices when it comes to data collection and annotation methods for Machine Learning datasets. This research paper systematically identifies and synthesizes such processes by examining existing literature on recommender systems. The review includes 100 most-cited papers from the most impactful venues within the Computing and Information Technology field. Multiple facets of the employed techniques are touched upon, such as reported human annotations and annotator diversity, label quality, and the public availability of training datasets. Recurrent use of just a few benchmark datasets, poor documentation practices, and reproducibility issues in experiments are some of the most striking findings uncovered by this study. A discussion is centered around the necessity of transitioning from reliance solely on algorithmic performance metrics in favor of prioritizing data quality and fit. Finally, valid concerns are raised when it comes to biases and socio-psychological factors inherent in the datasets, and further exploration of embedding these early in the design of ML models is suggested.

Web surveys have increasingly been used to collect data from respondents over the years. They offer several advantages compared to other methods of obtaining data. Researchers benefit from a broad demographic representation to make generalized conclusions, and satisfaction surveys allow employees to explain shortcomings or improvements anonymously. Both examples demand comprehensive information, thereby requiring a lengthy survey. However, dropout increases with the length of a survey, which is a big problem on web surveys as it decreases the statistical significance of the results. Proposed solutions, such as reducing the number of questions or rewarding respondents with an incentive, may not always be feasible due to the preciseness of information required or limited financial capabilities. Achievement primes have been shown to reduce dropout on short surveys targeting extrinsically motivated respondents without additional costs or the need to reduce survey length. As repeated exposure to primes reinforces the stimuli, long surveys may also benefit from achievement primes. In this study, respondents are exposed to a questionnaire of more than 15 minutes on health whilst working behind a computer containing either no prime, passive achievement primes, or active achievement primes. Besides extrinsically motivated respondents, recruited via the crowdworking platform Prolific, intrinsically motivated respondents are also targeted in this study, recruited via snowball sampling. Through a 2 times 3 factorial design, we discovered no statistical difference in dropout, perceived workload, and user engagement across the three questionnaire variants when evaluating intrinsically (N=88) and extrinsically motivated respondents (N=140) individually. By comparing intrinsically with extrinsically motivated respondents, we discovered extrinsically motivated respondents were more engaged and dropped out less.

The increasing dangers of unfairness in machine learning (ML) are becoming a frequent subject of discussion, both, in academia and popular media. Recent literature focused on introducing and assessing algorithmic solutions to bias in ML. However, there is a disconnect between these solutions and practitioners' needs. By interviewing 30 ML practitioners of diverse backgrounds across 16 countries, and presenting them with a simulated use case, our study aims to investigate common fairness practices among professionals and how these are influenced by their backgrounds. The results reveal a superlative disparity among academia and industry practitioners. We also identify different practices in fairness and data exploration stages, influenced by the educational background as well as the level of experience of practitioners. Our study also finds how demographics have an impact on several aspects, such as willingness to accept and support legal actions taken against ML discrimination. In accordance with our findings, we suggest several actions that can be taken to improve fairness solutions, and we also highlight future directions for fairness research that can cause a positive impact on the way fairness is perceived by practitioners.

Longer lifespans and an ageing population put tremendous pressure on the care of the elderly. With the technology of robotics breakthroughs, it appears that the use of robotics in elderly care is ready to take off. Interestingly, more and more robots are now being created with a human-like appearance and demeanour, yet this could lead to issues like inappropriate mental models of robots in the user's mind. On the other hand, due to their minimal communication challenges, robot-like robots may function effectively in the human-robot interaction. Up to now, far too little attention has been paid to the increasing trend of the human-like robot. This thesis compared the performance of these two types of robots, which may help to explain why the trend is the way it is. Two research questions are created to accomplish this goal: Which type of robot, robot-like or human-like, produces the most fluent interaction in the elderly care scenario? Does the robot-like or human-like robot bring about the best memory performance in the music listening activity? This thesis project aims to contribute to the development of the social robot in elderly care. In this thesis, a music assistance robot was developed, it can converse with the user while listening to music and recalling memories associated with the song. The elder's health and well being are benefited by this pastime. To address the question of which type of robot performs better, two types of robots, a human-like robot and a robot-like robot, were created to compare in terms of fluency in human-robot interactions and performance of memory recall during music listening activities. Four key elements: appearance, behaviour, voice and dialogue were used in the construction of two robot identities. Based on the robot we created, a joint music listening experiment was carried out. For evaluation, a questionnaire and video recordings of the interaction were both employed. This was tested with 30 volunteers (The young adults were chosen for the experiment due to COVID restrictions and time limitation), and it was discovered that the human-like robot performed better overall.

Cognitive Impairment, commonly termed as Dementia, affects a large number of older adults. People with dementia (PwD) experience cognitive decline that impacts their ability to perform daily activities and maintain social connections. The number of PwD is expected to rise, and unfortunately, there is still no cure. A number of non drug therapies such as music and art therapy are used to stimulate neurons and slow down the progress of the disease. When symptoms become severe, full-time care may be needed, requiring a move to a nursing home for formal care. However, studies show that interaction with peers is limited in a care home, leading to loneliness and reduced quality of life. Caregivers organize group activities like singing and baking to alleviate these feelings, but their busy schedule can limit the frequency of these exercises. This begs the question: Can a robot support caregivers in the provision of group activities in a care home? This would allow caregivers more time to attend to the personal needs of the residents. It not only has the potential to reduce workload of caregivers but also creates opportunities for residents to interact with each other. In this thesis, we propose the design of a robot which moderates a sing-along activity for a group of people living in a nursing home. An activity in the realm of music therapy is chosen since it is one of the non-drug therapies which has shown to have numerous benefits like improving memory recall, eliciting emotions and aiding relaxation. The robot session is curated to give opportunities for interaction with group members and with the robot too. While the song is playing and participants are singing along with the music, the robot performs engagement detection, and encourages those who do not seem involved. Multiple ways are proposed to detect engagement in HRI, such as emotion recognition and estimation of head pose. These approaches have limitations in the scope of this thesis, particularly because these methods are not designed for groups of older adults residing in a care home. Given these limitations and our focus on the target group, we propose a novel engagement detection methodology, which leverages the presence of a robot and assigns an active role to it. Instead of relying on conventional passive engagement detection methods, this approach uses an interactive probing technique. The robot prompts users to perform certain actions, and whether or not they respond to it is monitored using pose estimation. This is termed robot engagement. This approach is combined with activity engagement, which indicates whether the participant is singing or not. With this hybrid technique, we aim to increase the efficacy of engagement detection. If a participant is detected as disengaged, either in the activity or in the robot, we provide personalized encouragement and motivation by addressing participants by name. Along with this, the robot also compliments those who actively participate, motivating them further...

Cognitive Impairment, commonly termed as Dementia, affects a large number of older adults. People with dementia (PwD) experience cognitive decline that impacts their ability to perform daily activities and maintain social connections. The number of PwD is expected to rise, and unfortunately, there is still no cure. A number of non drug therapies such as music and art therapy are used to stimulate neurons and slow down the progress of the disease. When symptoms become severe, full-time care may be needed, requiring a move to a nursing home for formal care. However, studies show that interaction with peers is limited in a care home, leading to loneliness and reduced quality of life. Caregivers organize group activities like singing and baking to alleviate these feelings, but their busy schedule can limit the frequency of these exercises. This begs the question: Can a robot support caregivers in the provision of group activities in a care home? This would allow caregivers more time to attend to the personal needs of the residents. It not only has the potential to reduce workload of caregivers but also creates opportunities for residents to interact with each other. In this thesis, we propose the design of a robot which moderates a sing-along activity for a group of people living in a nursing home. An activity in the realm of music therapy is chosen since it is one of the non-drug therapies which has shown to have numerous benefits like improving memory recall, eliciting emotions and aiding relaxation. The robot session is curated to give opportunities for interaction with group members and with the robot too. While the song is playing and participants are singing along with the music, the robot performs engagement detection, and encourages those who do not seem involved. Multiple ways are proposed to detect engagement in HRI, such as emotion recognition and estimation of head pose. These approaches have limitations in the scope of this thesis, particularly because these methods are not designed for groups of older adults residing in a care home. Given these limitations and our focus on the target group, we propose a novel engagement detection methodology, which leverages the presence of a robot and assigns an active role to it. Instead of relying on conventional passive engagement detection methods, this approach uses an interactive probing technique. The robot prompts users to perform certain actions, and whether or not they respond to it is monitored using pose estimation. This is termed robot engagement. This approach is combined with activity engagement, which indicates whether the participant is singing or not. With this hybrid technique, we aim to increase the efficacy of engagement detection. If a participant is detected as disengaged, either in the activity or in the robot, we provide personalized encouragement and motivation by addressing participants by name. Along with this, the robot also compliments those who actively participate, motivating them further...

Machine learning is still one of the most rapidly growing fields, and is used in a variety of different sectors such as education, healthcare, financial modeling etc(Jordan and Mitchell 2015). However, along with this demand for machine learning algorithms, there comes a need for ensuring that these algorithms are fair and contain little to no bias. Tools like Fairlearn and AI Fairness 360(AIF360) allows developers and data scientists to examine their codebase according to specified fairness metrics and mitigate any fairness related issues. This study aims to determine how practitioners use the separate toolkits and whether their practices are differ by the toolkit they choose to use. To do this, we conducted 29 think-aloud interviews with industry practitioners to understand how they would use Fairlearn and AIF360 in practice. The results show that fairness is a socio-technical challenge. While the toolkit does allow for participants to be structured in their approach, and raise awareness for fairness related harms, at the end of the day the fairness toolkit only provides technical help to find harms that the individual was already aware about. Based on the findings, we then suggest the design for a fairness toolkit that can help practitioners approach fairness in the most ideal manner. This toolkit would include a way to have interdisciplinary collaboration, have a larger focus on explainability, and give clear guidance to its users regarding fairness related harms.

The ability to identify and mitigate various risks and harms of using Machine Learning models in industry is an essential task. Specifically because these may produce harmful outcomes for stakeholders, including unfair or discriminatory results. Due to this there has been substantial research into the concepts of fairness and its metrics, bias and its mitigation, and algorithmic harms and their sources. Various toolkits have been created to guide practitioners to reflect on these topics and provide suggestions on algorithmic solutions to mitigate these risks. However, it is not yet known how widely these toolkits are used and how they are perceived in terms of usefulness. In this project, practitioners were interviewed in order to determine to what extend do envisioned practices of practitioners without experience with fairness toolkits differ from those with the experience. The two toolkits considered were the IBM AI Fairness360 and Microsoft FairLearn. The data collected from the interviews suggests that there could be fewer differences in practices of practitioners with experience and without experience with toolkits, than those with training or work roles in ethics and fairness in ML and those without. This suggests that experience the toolkit itself is not indicative of a more thorough approach to identifying and mitigating harms in fair Machine Learning.

Research that focuses on examining software bugs is critical when developing tools for preventing and for fixing software issues. Previous work in this area has explored other types of systems, such as bugs of compilers and security issues stemming from open source systems hosted on public repositories. This paper explores the bugs within the Ansible software provisioning and configuration management system. The main question this paper seeks to answer is "What common patterns can be extracted from the bugs found and what are the root causes, symptoms, triggers, system-dependence factors, fixes, and the impact of the most frequent types of bugs in the Ansible configuration management system". This study defines a data pipeline and custom tools to extract and analyze 100 Ansible bugs. Common classifications are determined, and the bugs are manually classified, revealing common patters within the bugs. Insights are drawn from the aggregated data, and recommendations are made for addressing bug-prone areas of execution and connectivity components, and expanding the test suite with input fuzzing and a genetic algorithms test solution, in order to improve the overall code quality of the Ansible code base.

Several input types have been developed in different technological landscapes like crowdsourcing and conversational agents. However, sign language remains one of the input types that has not been looked upon. Although numerous amount of people around the world use sign language as their primary language, there have not been many efforts to include them in these technological landscapes. In this thesis, we hope to draw attention to and take a step towards the inclusion of deaf and mute people in microtask crowdsourcing. We identify some of the existing technical and research gaps in the current architectures for Sign Language Recognition/Translation in a real-time setting. Next, we determine various microtasks which can be adapted to use sign language as input, keeping in mind the challenges it introduces. We, then, investigate the effectiveness of a system that uses sign language as input by building a web application - SignUpCrowd - for microtask crowdsourcing, namely Visual Question Answering and Tweet Sentiment Analysis tasks, and comparing it with already prevalent input types such as text and click. This comparison with different popular input types will help understand how much of a difference there is for sign language as input. In addition, it will also show the preference of input types for the particular microtasks. For this, we developed three web applications with different input types and conducted a between-subject experimental study on Prolific wherein a number of workers (N=240) were asked to perform the above-mentioned tasks using sign language, text, and click input. Our results indicate that, in terms of task completion time and task accuracy, sign language as an input modality in microtask crowdsourcing is not significantly different from other, commonly used, input types. We also noticed that people's input type preference for the given microtasks for sign language was more than text input. Although people with no knowledge of sign language found it difficult, this input modality aims at a different target audience. This shows us that there is scope for sign language as an input type for microtask crowdsourcing among people, and paves the way for more efforts for the introduction of sign language in real-world applications.