As the amount of information available in the world grows, Information Retrieval (IR) systems have become an integral part of day to day life. They determine what subset of the large pool of information is shown to people. IR algorithms determine which items should be returned in response to a query and rank the results in a ranked list.
Recently, concerns about the fairness of IR algorithms have surfaced. In particular, research is being done into whether IR algorithms are fair to producers, people or organizations that provide the items that are retrieved by IR algorithms. The higher an item is in the ranked list, the more attention it receives from users. This attention translates to benefits for the producers, e.g. fame or financial compensation.
In this thesis we investigate the fairness of IR algorithms in terms of a specific measure for provider fairness: the Expected Exposure Loss (EEL). This measure measures whether the providers of equally relevant items receive the same amount of attention in expectation. EEL was first proposed as part of the 2020 TREC Fair Ranking track (FAIR-TREC), which also provided a matching dataset. We investigate for two IR systems whether they achieve fairness on this dataset. We conduct a failure analysis and propose improvements for both systems.
We find that for a system that always returns the same ranking it is not useful to improve its accuracy, but rather that it benefits most from fairness-aware post-processing. By contrast, a fairness-aware systems does benefit from a higher accuracy, since EEL requires that equally relevant items are treated the same. We note that the generalizability of our investigation is limited due to the small size of the FAIR-TREC 2020 dataset and recommend that a larger dataset be made available. ... Read more

Performing user interface studies and logging user interactions is commonplace in the Interactive Information Retrieval field (IIR). As a result, the LogUI framework was developed to make logging such interactions an easier task. However, this framework does not come with any visualization tool, requiring the researcher to analyze their data separately. We present a generalized data dashboard that allows the researcher to perform basic exploratory analysis on their data, without needing to parse their logging data themselves. We gathered a set of important logging metrics from existing user interface experiments and based our dashboard design on these metrics. Finally, we verified the usability and effectiveness of the dashboard by interviewing a set of IIR researchers and show that we developed an effective dashboard. ... Read more

Logging user interactions allows researchers to study user behavior in web applications. The LogUI framework can be used to log user interactions on virtually any web application. This framework misses two vital features, a screen capture feature and a dashboard for statistical data analysis and visualization for use in web-based experiments.Here we have implemented this dashboard and screen capture feature as an addition to LogUI. We cover tools for screen recording and methods to limit bandwidth usage when sending these recordings between client and server. For the dashboard, we worked in cooperation with interactive information retrieval (IIR) researchers working with the open-source collaborative search system SearchX. Using their input, we determined metrics, visuals, and features to be included in the dashboard.
Our results demonstrate that recording a single browser tab as opposed to the entire screen or a browser window is more bandwidth-efficient when sending recordings from client to server. Using the H.264 video format results in much better compression compared to other formats available. A low bitrate setting also results in low bandwidth usage, but this setting should be set with care. The dashboard intends to prevent researchers from having to implement their own scripts for data analysis. Its main focus is comparing aggregations of sessions, but it does allow the analysis of single sessions. Including metrics like dwell time, time between queries, session duration, and event occurrences, as well as visuals like box plots, time series plots, and an event timeline is highly beneficial for IIR researchers using LogUI. Being able to relate logs to their occurrence in a screen recording is also advantageous. In an evaluation session, the researchers were able to complete a series of tasks with little difficulty on average. Some features, like filtering specific sessions, or metrics for the visuals, are not fully comprehensible. However, in its evaluation, five out of five researchers noted that they would use the dashboard in their future research. This shows that the included metrics, visuals, and features have been correctly selected. ... Read more

Collaborative search is "the retrieval and sensemaking of information by multiple users with the same information need". The experience of collaborative search can be improved by employing a chatbot which is "a program designed to counterfeit a smart communication". Chatbot makes it easier for the users to be aware of each other's activities and to share, communicate and reach consensus during collaborative search and makes it more enjoyable. However, the prior research has been limited by using a single-user web search engine and a dedicated messaging platform. It is unknown whether the observed benefits are maintained in a collaborative search system where several collaborative tools exist, such as shared query history, bookmark, and built-in messaging capability. Hence, here we aim to explore how chatbot affects users' behavior on a collaborative search system and users' perceptions of the chatbot. We implemented ChatX, a chatbot agent that monitors the group chat and guides users through the search tasks. To evaluate it, we conducted a user study with X users in the domain of restaurant recommendations and local attraction searching. We found that ChatX does not significantly reduce users' search effort, but it improves the overall collaborative search experience and gives the user a highly favorable impression. ... Read more

Items that a user can see when he uses the general result page of a modern search engine can be categorized as verticals. Some examples of verticals are images, videos, news, shopping. Heterogeneous search engine result pages encompass result pages that contain results from different verticals. It is widely used and has been proven to improve the user experience over the result pages that only contain a list of websites. Different verticals are appropriate for each query. We study how to define, develop, and evaluate a vertical selection model, that for a query selects and presents the appropriate verticals. We give an approach for collecting a corpus of documents that represent different verticals. Later corpus documents are used as training data for query result classification. Features were extracted from the documents to train a classifier. The model that uses the Random Forest classifier and features extracted from the query itself achieved an f-score of 0.4921 on the TREC 2014 dataset. The score and the analysis of the results show that the proposed vertical selection methodology is viable. To better capture the difference between documents in different verticals, the corpus collection approach should be improved. ... Read more

Search engine Entity Cards(ECs) display conciseinformation from the web about a topic or subjectin response to a user query. The topic or subjectcan be a person, an organization etc. and is referredto as an “Entity”. The specific topic under researchis how to determine which entity is most relevantfor the query in terms of helping the user find theinformation he/she is looking for. and what infor-mation about the chosen entity to display to answerthe query. The information can be in the form ofbut it not limited to text, images and hyperlinks.Research into the concepts of EC focuses on differ-ent components of the EC widget for example en-tity linking, tagging, extraction and fact summarygeneration. In the developed “EC algorithm” theseconcepts are combined into an implementation ofan Entity Card widget and then evaluated. The ECalgorithm utilizes tools such as DBPedia, DBPediaSpotlight and the Bing Web Search API to gener-ate an entity ranking for a query. The results ofevaluating the top ranked entity imply that the ECalgorithm retrieve on average a slightly to moder-ately relevant entity to the user. The fact retrievalalgorithm had predictably worse results given thecomplexity of finding truly relevant facts about en-tities. ... Read more

In order determine when we can show direct answer module to user queries in web search engine, an independent classifier is designed in this study to assess the answerability of each user query. Real user queries are sampled from MS MARCO Question Answering and Natural Langauge Generation dataset \cite{MSMARCO} and manually labelled with query answerability to train and evaluate the classifier. As a result, the XGboost model has an overall better performance than the random forest model with prediction accuracy score 0.83 and F1 score 0.89. Once the classifier determines the user query is answerable, a MRC model may be used to find the direct answer within provided passages. Else, no direct answer shall be provided to this query. ... Read more

Rotterdam Werkt! is a network of fourteen organizations in the Rotterdam area in the Netherlands. Their goal is to increase labor mobility between these organizations through sharing vacancies, exchanging employees and partaking in joint projects. Rotterdam Werkt! has tasked us with creating a central platform on which all vacancies are automatically combined from the websites of all the organizations in the network. The two main challenges of the project were to gather the vacancies from all the organizations affiliated with Rotterdam Werkt! and allow their recruiters to search and filter through them. This meant that a significant amount of research needed to be done in order to find a suitable scraping tool as well as a suitable search engine. Whilst gathering the vacancies, we ran into the problem that each website was significantly different in the way it is rendered. Furthermore, we also needed to categorize the data correctly such that it becomes searchable in the search engine. Lastly, the retrieval function needed to be optimized such that the most relevant vacancies would be returned for a given query. In order to assess whether recruiters could use the search engine effectively in practice, an evaluation of the effectiveness of the search engine was done. Three retrieval functions were compared based on a significance test of several effectiveness measures that indicate to what extent a retrieval function is able to retrieve relevant documents, or in this case, vacancies. Out of the three, the retrieval function that scored the highest was chosen to be used in the platform, so that recruiters will have a bigger chance to find the vacancies they will be looking for. In the end, we consider our project to be a success. We managed to scrape all vacancies from all the websites of the organizations in Rotterdam Werkt! and to combine these on a centralized platform. Furthermore, the search engine evaluation allowed us to select the best vacancy retrieval function out of the three evaluated retrieval functions. However, more work can still be put into evaluating the search engine in the future by testing more retrieval functions based on more vacancy data, so that the search functionality can be further improved.
... Read more

With the world in grasp of the COVID-19 pandemic, models predicting the spread of the virus can give indications to what extent a country is controlling the pandemic. Policymakers can decide to install so-called mitigation strategies to limit the spread of the virus. To aid the decision-making process, this report describes how a web application was created that is capable of visualising predictions on the future course of the virus spread in the Netherlands. Furthermore, the application allows for changing the spread rate of the virus to simulate both mitigation and exit strategies. Research has been conducted on how we can combine predictions and simulations of mitigation strategies in a single visual solution, in order to aid policymakers. Existing products were analysed in order to get a better understanding of the users’ wishes. Design goals were established which have been taken into account when designing and building the software. Furthermore, suitable languages and frameworks for the implementation were chosen. We have created a tool which both implements a prediction algorithm and visualises the outcomes of this algorithm in a web application. First, a visual design of the product was created after which an accompanying software architecture was established. This design and architecture were then implemented and tested accordingly. Most of the conducted tests were unit tests, but also user tests were performed. During the implementation phase, potential ethical consequences were considered and handled accordingly ... Read more

During conversational information retrieval, a user engages in a dialogue interaction with a search system in order to satisfy an information need. A profound understanding of the way in which users formulate and reformulate messages during this dialogue interaction, will aid the development and optimization of conversational search systems. This research analyses what query reformulation types are frequently used, and looks at how this differs between fact finding and information gathering search tasks. Existing research on query reformulation mainly focuses on traditional IR systems. The little research that has been conducted in a conversational context is based on interactions between humans, rather than incorporating a search engine. We are interested in conversational query reformulation in a text-based interface, using a web-based search engine. To this end, preliminary insights of an empirical user study are presented. On the basis of its results, a taxonomy of query reformulation types is defined. Additionally, significant differences are found between how fact finding and information gathering messages reformulate queries. These results contribute to a better understanding of the conversational search dialogue, which aids the further research and development of conversational search systems. ... Read more

Conversational search systems have recently gained popularity due to their effectiveness in performing small tasks and answering factoid questions. However, under complex search scenarios, these systems fail and fall back to techniques used by traditional search engines. As tasks requiring user learning are inherently complex, user knowledge is a powerful indicator of system performance under complex search and user ability to successfully interact with the information content. We present a study in which user knowledge gain and query formulation is explored in both the traditional web search and conversational search formats. Through crowd sourcing, 50 participants were recruited and conducted complex search sessions on either the traditional or conversational search medium. Through the use of a knowledge test given to the participant both before and after the search session, knowledge gain was determined. Furthermore, session queries and timestamps were tracked. It is found that participants have a significantly higher knowledge gain in traditional search, while conversational search sessions tended to have a higher number of queries and average query length. ... Read more

This paper delved into the effects of domain expertise on a user's conversational search, because as the use and acceptance of voice assistants increase the need for conversational search agent that can accommodate to a human characteristic such as domain expertise. Accommodating to disadvantaged users of web search as earlier works showed that users with low literacy and low spatial visualization abilities are strongly affected in their searches compared to users who do not suffer from these impairments. Prior research into domain expertise demonstrated the influence it has had on the querying behavior of users in web search. They found that domain experts included more domain specific jargon in their messages, made longer queries and spent less time per search task. This paper examined these findings in a conversational search setting. Contrary to these findings, no significant relation between the domain expertise level and any of these results could be established. However, conducting the experiment to assess these findings has provided insight into how users respond to a conversational search study such as this one. ... Read more

While voice assistants have exploded in popularityover the last decade, they still have many issues.Among these is the issue of result presentation:how do you speak results to the user? Priorresearch has investigated how cognitive loadrelates to result presentation and other methodsfor result presentation, but highlighted a lackof research about result presentation over speechand called for further investigation. [12] Thisresearch answered that call, and investigated the research question How can web search results bepresented over a speech-only channel such thatthe users’ cognitive load is minimized?Wetested three methods: 1) one-source, one-shotresponses, 2) question-and-answer responses, and3) multi-source, one-shot responses. In all cases,single-source responses were preferred to multi-source responses. Of the two types of single-source responses tested (question-and-answer styleresponses and one-shot responses), both types hada similar average score. Our recommendationto developers of conversational search technologywould be to allow user choice between these tworesponse types. We would only recommend multi-source responses as a fall-back option in the caseof a low estimated probability of result relevance. ... Read more

This paper shows the influence of multitasking on the usage of a voice assistant. Voice assistants allow users to input queries over a speech-only channel, and as a result they do not require the same attention as a traditional search engine. Existing research describes the effects of the use of a voice assistant on driving and other demanding activities, however, there is no research that describes how that demanding activity influences the usage of the voice assistant. To research these effects, three sub questions have been constructed and answered. These tackled three characteristics to describe the usage: the query formulation, the knowledge gain and the user experience. To answer these questions a user study was conducted where the participants used a voice assistant in three situations. Two of these situations included a distraction in the form of a game. We found that the presence of dual tasking results in shorter queries. We also found that a higher intensity of the session can decrease the knowledge gain. ... Read more

Healthcare industry is an ever-emerging field in the 21st century. The statistics from Centers for Medicare services (CMS) [15] website shows that in 2017, in USA, the healthcare industry has invested USD 7.4 billion for research collaborations with physicians. These research collaborations in CMS is spread across multiple facets ranging from contributing towards research, developing new products, running clinical trials, royalty, licences, patents, providing innovative ideas etc. In this thesis, we make an assumption that, a relationship between investment made by the healthcare company and the research profile of a physician exists. We aim to answer, what could possibly be the relation between payments made by the healthcare company; on the physicians and the research profile of the physicians. The research profile of a physician includes factors like h-index, years of research experience, citation count, physician citation network, etc. To validate this relationship we use the data corresponding to returns of the healthcare company. Some of the measures of returns, from a research collaboration between physicians, include, innovation, good will, fame, market share, Return on Investment (ROI). We choose ROI, as a measure of return, due to the availability of data and to determine the relationships mentioned above.

To understand the above mentioned relationship, we explore two types of relationships, i.e., direct and indirect relationship. In the direct relationship, we use multiple regression model to understand the direct relationship between the research payments and the research profile of the physicians, by making an assumption that the research profile of the physician describes the research quality of the physician. In the indirect relationship, we make use of a weighted physician co-author citation network, to investigate the relationship between his/her co-author interactions and the research payments he/she received from the healthcare company. To accomplish this, we developed a spreading process that models influence diffusion in a physician citation network. The diffusion of influence is dependent on the topological property of the node in the network.

Our models are an exemplification of the direct and indirect relationships, which exists in the real world. To evaluate our models, we use metrics such as coefficient of determination, Pearson correlation coefficient and Spearman's rank correlation. Once the models were evaluated, we inferred that the model for indirect relationship, explains the relationship between research profile, investments, and return 96.3% more than the model for direct relationship. We also perform a deep analysis, by investigating the nature of the distributions of the variables and scatter plots to understand the relationship between the variables used in understanding the direct and indirect relationship. Lastly, we propose two different redistribution methods, where the original payments made to physicians are redistributed to a potential group of physicians in the physician citation network. These potential physicians are identified based on their topological property. In consequence, our redistribution methods may inspire the healthcare companies, to design their future investments made to physicians.
... Read more

Weak baselines have been present in Information Retrieval (IR) for
decades. They have been associated with IR progress stagnation, baseline
selection bias to publish results more readily, and models’ effectiveness
reproducibility issues that hinder the validation of results by independent
research teams. Weak baselines have been studied by the IR community;
however, the focus has been almost exclusive on ad-hoc retrieval, the most
popular IR task, leaving outside other IR tasks and datasets recently de-
veloped. Current deep neural IR research is particularly vulnerable to the
issues with weak baselines due to the hype surrounding deep learning.
In this thesis we investigate the cases of weak baselines in ad-hoc
retrieval and question answering (QA), two representative IR tasks among
13 cases of weak baselines we found in current deep neural IR research from
EMNLP 2018 conference. In particular, we study whether the recently
introduced deep neural IR models are actually significantly more effective
than the reported IR baselines or than LambdaMART, the Learning to
Rank (LTR) model we propose plus hyperparameter optimization (HPO).
We also benchmark two HPO methods: RS and BOHB, to determine which
method is more efficient to retrieve a good hyperparameter configuration.
Throughout our experiments we show that the effectiveness of the
novel deep neural IR models can be difficult to replicate, it might be lower
than reported, and that it is not necessarily significantly higher than the
baseliness. Furthermore, we demonstrate that BOHB is more efficient
than RS, but the HPO process not always improves the effectiveness of
LambdaMART significantly. ... Read more

Switches that can be (re)programmed through the network programming language P4 are able to completely change – even while in the field – the way they process packets. While powerful, P4 code is inherently static, as it is written and installed to accommodate a particular network requirement. Writing new P4 code each time new requirements arise may be complex and limits our agility to deal with changes in network traffic and services. To solve that problem, we present two frameworks within this work: P4I/O and GP4P4. P4I/O is a new approach to data-plane programmability based on the philosophy of Intent-Based Networking. P4I/O provides an intent-driven interface that can be used to install and/or remove P4 programs on the switches when needed and which is easy to use. In particular, to realize P4I/O, we (1) describe an extensible Intent Definition Language (IDL), (2) create a repository of P4 code templates, which are parsed and merged based on the intents, (3) provide a technique to realize the resulting P4 program in a programmable switch, while accommodating intent modifications at any time, and finally (4) implement a proof-of-concept to demonstrate that intent modifications can be done on-the-fly. Our second framework, GP4P4, is a genetic programming approach able to autonomously generate programs for P4-programmable switches directly from network intents. We demonstrate that GP4P4 is able to generate various network functions in up to a few minutes; an important first step towards realizing the vision of ‘Self-Driving’ networks. ... Read more

With the increasing popularity of mobile and voice-assisted, extracting short and precise answer passages to open-domain questions is becoming an increasingly important information retrieval (IR) task. The recently released large-scale corpus for answer passage retrieval—WikiPassageQA—was shown to be challenging for both traditional retrieval models and neural architectures. One of the classic approaches to improving retrieval effectiveness across tasks is automatic query expansion (QE). QE is the process of reformulating a user’s query by adding more terms with the goal of retrieving more relevant information. Word embeddings are commonly employed to obtain QE terms by taking advantage of the low dimensional semantic space formed by these embeddings.
Recently, Diaz et al. showed that QE using word embeddings trained on a local query-specific corpus performed better than embeddings that were trained on an entire global corpus for document ranking tasks. We aim to examine the effectiveness of QE, specifically using locally-trained word embeddings, in this new context of answer passage retrieval. Additionally, a query-specific corpus can be small in size with limited vocabulary which forms a challenge for training word embedding models. Since the extent to which limited vocabulary influences the semantic information captured by word embeddings is relatively unexplored, we compare two word embedding models—CBOW and IWE—in this thesis. Having the same underlying training philosophy, the IWE model differs from CBOW in two aspects—it incorporates sub- word information of words and uses a convolutional neural network to learn context representation.
Our results corroborate the findings of Diaz et al.—query-specific data is also beneficial in the task of retrieving passages to open-domain questions. Word embeddings trained on a global corpus fail to capture the nuances of query-specific language present in the answer passages. We also found out that IWE word embeddings capture more semantic information than CBOW word embeddings when trained on local data with a limited vocabulary. Our experiments show that both the IWE model components contribute to the improved quality of word embeddings and consequently better QE terms. Our work can be extended by using the same methodology in other domains or by using different word embedding models to obtain QE terms. The insights from our thesis can help researchers to make an informed decision while choosing word embedding models and training data for their IR and natural language understanding tasks. ... Read more

Open-domain question answering (QA) is an important step in Artificial Intelligence and its ultimate goal is to build a QA system that can answer any question posed by humans. The majority of the open-domain QA system is the retrieval-based open-domain QA system, which enables the retrieval component to retrieve relevant documents from a large-scale knowledge source to a question and the answer extraction component to extract the answer to this question based on retrieved documents. As the techniques of Deep Learning progressing significantly, many researchers tried to apply the neural reading comprehension (RC) model to serve the answer extraction component of the open-domain QA system. However, the performance of the neural RC model in open-domain QA is considerably worse than the performance of it in RC-style QA. Therefore, many works have focused on the neural RC model for addressing the performance gap, whereas the retrieval component of the open-domain QA system lacks equivalent attention. Some researchers have built the neural network based information retrieval (IR) models, but currently, it is still difficult for these neural IR models to directly retrieve documents from a large-scale knowledge source in open-domain QA. Hence, many works attempted to use neural IR models for re-ranking documents retrieved by the traditional but efficient IR models (e.g., TF-IDF, BM25) in open-domain QA. However, these works did not analyze the impact of different questions of QA datasets on traditional IR models. Thus, this research gap is the focus in this thesis. We conduct error analyses of questions of different QA datasets to figure out the error types of questions that have a negative impact on the traditional IR models. From the error analysis, we learn that different QA datasets have different impacts on the traditional IR models and are differently hard to be dealt with by the traditional IR models. Therefore, we propose hypotheses that might mitigate the negative impact of the error types of questions that are relatively harder to be handled by the traditional IR models. Furthermore, we perform experiments based on the methodologies that implement our hypotheses for figuring out the validity of these hypotheses. In conclusion, we believe that our work is a step forward to obtaining more insights into the retrieval component of the open-domain QA system and will contribute to the development of the retrieval component for a better open-domain QA system. Moreover, our work can give our users guidance on how to issue a more suitable question that can be processed by the open-domain QA system for giving a more accurate and better answer. ... Read more

After surpassing human performance in the fields of Computer Vision, Speech Recognition and NLP, deep learning has been gaining scientific ground in IR. In spite of the sheer amount of publications that have proposed so-called neural IR approaches over the past decade, the field has not achieved the kind of progress seen in related fields. Over the past year or so, works have begun to solve the issues that complicate the progress of neural applications in IR. Among those issues we can find the lack of approaches to interpret and analyze neural IR models, which is addressed in this thesis. We propose a novel approach to diagnose retrieval models that is rooted in the axiomatic approach to IR. Axioms encapsulate search heuristics that are expressed as constraints on retrieval functions. Existing axiomatic approaches have provided fruitful analyses of traditional IR models but are no longer viable to study neural IR models. Building forth on these approaches, we propose a novel approach to empirically analyze retrieval functions, suitable for neural models. Based on inspirations from the NLP and Computer Vision communities, we use model-agnostic diagnostic datasets in order to determine what kind of search heuristics models are able to learn. Since the creation of diagnostic datasets does not require a labeled dataset, we can apply the proposed pipeline to almost any dataset containing queries and documents. We have shown for four specific axioms how to extend and relax them, in order to make them fit for obtaining diagnostic datasets. We have applied our diagnostic dataset creation pipeline to the WikiPassageQA and MSMarco corpora and evaluated three traditional baselines and six neural models. Our experiments on the WikiPassageQA dataset show that the proposed approach can indeed diagnose strengths and weaknesses of neural models. However, our experiments on the MSMarco dataset show that an axiomatic analysis based on the four axioms does not always diagnose factors that incur retrieval effectiveness. An interesting direction for future work is therefore to include more axioms in the diagnostic approach. As possible extensions of the work carried out in this thesis, several roads of future work have been proposed. Among them, we can find reproducing experiments on other neural toolkits and employing the methodology on different IR tasks, but also researching the validity of axioms and adopting a specialized metric for axiomatic performance. We furthermore identified various opportunities to use diagnostic datasets beyond diagnosing neural models. Concluding, we believe that the axiomatic approach to diagnosing neural IR models presented in this work is a step forward to gaining valuable insights into the black boxes that deep models are generally considered to be. We hope our work may prove a fruitful resource for analysis in the field of neural IR on the road towards achieving superior performance without losing sight of a better fundamental understanding of IR. ... Read more