Remote Work Aided by Conversational Agents

Due to the coronavirus pandemic, remote work from home has rapidly become a necessity around the world, drastically changing the potential landscape for the future of work. Over the last couple of decades, microtask crowdsourcing has emerged as a viable means of carrying out remote online work to earn one’s living — an alternative to traditional...

This Is Not What We Ordered: Exploring Why Biased Search Result Rankings Affect User Attitudes on Debated Topics

In web search on debated topics, algorithmic and cognitive biases strongly influence how users consume and process information. Recent research has shown that this can lead to a search engine manipulation effect (SEME): when search result rankings are biased towards a particular viewpoint, users tend to adopt this favored viewpoint. To better...

Perspective: Leveraging Human Understanding for Identifying and Characterizing Image Atypicality

High-quality data plays a vital role in developing reliable image classification models. Despite that, what makes an image difficult to classify remains an unstudied topic. This paper provides a first-of-its-kind, model-agnostic characterization of image atypicality based on human understanding. We consider the setting of image classification...

Conversational crowdsourcing

The trend of remote work leads to the prosperity of crowdsourcing marketplaces. In crowdsourcing marketplaces, online workers can select their preferable tasks and then complete them to get paid, while requesters design and publish tasks to acquire their desirable data. The standard user interface of the crowdsourcing task is the web page,...

Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics

The way pages are ranked in search results influences whether the users of search engines are exposed to more homogeneous, or rather to more diverse viewpoints. However, this viewpoint diversity is not trivial to assess. In this paper we use existing and novel ranking fairness metrics to evaluate viewpoint diversity in search result rankings. We...

What Should You Know? A Human-In-the-Loop Approach to Unknown Unknowns Characterization in Image Recognition

Unknown unknowns represent a major challenge in reliable image recognition. Existing methods mainly focus on unknown unknowns identification, leveraging human intelligence to gather images that are potentially difficult for the machine. To drive a deeper understanding of unknown unknowns and more effective identification and treatment, this...

Conversational interfaces for search as learning

Searching the web to learn new things or gain knowledge has become a common activity. Recent advances in conversational user interfaces have led to a new research opportunity - that of analyzing the potential of conversational interfaces in improving the effectiveness of search as learning (SAL). Addressing this knowledge gap, in this...

Just the right mood for HIT!: Analyzing the role of worker moods in conversational microtask crowdsourcing

Conversational agents are playing an increasingly important role in providing users with natural communication environments, improving outcomes in a variety of domains in human-computer interaction. Crowdsourcing marketplaces are simultaneously flourishing, and it has never been easier to acquire large-scale human input from online workers....

Improving Worker Engagement Through Conversational Microtask Crowdsourcing

The rise in popularity of conversational agents has enabled humans to interact with machines more naturally. Recent work has shown that crowd workers in microtask marketplaces can complete a variety of human intelligence tasks (HITs) using conversational interfaces with similar output quality compared to the traditional Web interfaces. In...

Towards Memorable Information Retrieval

Information overload is a problem many of us can relate to nowadays. The deluge of user generated content on the Internet, and the easy accessibility to a vast amount of data compounds the problem of remembering and retaining information that is consumed. To make information consumed more memorable, strategies such as note-taking have been found...

TickTalkTurk: Conversational crowdsourcing made easy

This demo presents TickTalkTurk, a tool that can assist task requesters in quickly deploying crowdsourcing tasks in a customizable conversational worker interface. The conversational worker interface can convey task instructions, deploy microtasks, and gather worker input in a dialogue-based workflow. The interface is implemented as a Web...

Great Chain of Agents: The Role of Metaphorical Representation of Agents in Conversational Crowdsourcing

Conversational agents are being widely adopted across several domains to serve a variety of purposes ranging from providing intelligent assistance to companionship. Recent literature has shown that users develop intuitive folk theories and a metaphorical understanding of conversational agents (CAs) due to the lack of a mental model of the...

Chatterbox: Conversational Interfaces for Microtask Crowdsourcing

Conversational interfaces can facilitate human-computer interactions. Whether or not conversational interfaces can improve worker experience and work quality in crowdsourcing marketplaces has remained unanswered. We investigate the suitability of text-based conversational interfaces for microtask crowdsourcing. We designed a rigorous...

Gender Choices of Conversational Agent: How Today’s Practice Can Shape Tomorrow’s Values

When creating conversational agents, designers have to make decisions about the way the agents present themselves. In this position paper, we identify and synthesize ethical dilemmas that conversational interface designers and researchers face around gender of conversational agents. First, we identify three layers that cause tension in designing...

Responsible Opinion Formation on Debated Topics in Web Search

Web search has evolved into a platform people rely on for opinion formation on debated topics. Yet, pursuing this search intent can carry serious consequences for individuals and society and involves a high risk of biases. We argue that web search can and should empower users to form opinions responsibly and that the information retrieval...