HybridEval: A Human-AI Collaborative Approach for Evaluating Design Ideas at Scale

Evaluating design ideas is necessary to predict their success and assess their impact early on in the process. Existing methods rely either on metrics computed by systems that are effective but subject to errors and bias, or experts' ratings, which are accurate but expensive and long to collect. Crowdsourcing offers a compelling way to...

To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systems

Powerful predictive AI systems have demonstrated great potential in augmenting human decision-making. Recent empirical work has argued that the vision for optimal human-AI collaboration requires ‘appropriate reliance’ of humans on AI systems. However, accurately estimating the trustworthiness of AI advice at the instance level is quite...

Explainable Artificial Intelligence for human-AI collaboration

As a society, we have come to notice the influence and impact Artificially Intelligent (AI) agents have on the way we live our lives. For these AI agents to support us both effectively and responsibly, we require an understanding on how they make decisions and what the consequences are of these decisions. The research _field of Explainable...

Value-Sensitive Rejection of Machine Learning Predictions for Hate Speech Detection

Hate speech detection on social media platforms remains a challenging task. Manual moderation by humans is the most reliable but infeasible, and machine learning models for detecting hate speech are scalable but unreliable as they often perform poorly on unseen data. Therefore, human-AI collaborative systems, in which we combine the strengths of...

Fictional Co-Play for Human-Agent Collaboration: Evaluating state-of-the-art reinforcement learning technique for adaptability to human collaborators

A longstanding problem in the area of reinforcement learning is human-agent col- laboration. As past research indicates that RL agents undergo a distributional shift when they start collaborating with human beings, the goal is to create agents that can adapt. We build upon research using the two-player Overcooked environment to repro- duce a...

Towards Trust in Human-AI Teams

Human-AI teams require trust to operate efficiently and solve certain tasks like search & rescue. Trustworthiness is measured using the ABI model; Ability, Benevolence and Integrity. This research paper tries to observe the effect a conflicting robot has on the human trustworthiness. The hypothesis we try to test is: “human trustworthiness...

Design patterns for human-AI co-learning: A wizard-of-Oz evaluation in an urban-search-and-rescue task

The rapid advancement of technology empowered by artificial intelligence is believed to intensify the collaboration between humans and AI as team partners. Successful collaboration requires partners to learn about each other and about the task. This human-AI co-learning can be achieved by presenting situations that enable partners to share...

Peer grading the peer reviews: A dual-role approach for lightening the scholarly paper review process

Scientific peer review is pivotal to maintain quality standards for academic publication. The effectiveness of the reviewing process is currently being challenged by the rapid increase of paper submissions in various conferences. Those venues need to recruit a large number of reviewers of different levels of expertise and background. The...

Hybrid collective intelligence in a human–AI society

Within current debates about the future impact of Artificial Intelligence (AI) on human society, roughly three different perspectives can be recognised: (1) the technology-centric perspective, claiming that AI will soon outperform humankind in all areas, and that the primary threat for humankind is superintelligence; (2) the human-centric...