With the increasing popularity of mobile and voice-assisted, extracting short and precise answer passages to open-domain questions is becoming an increasingly important information retrieval (IR) task. The recently released large-scale corpus for answer passage retrieval—WikiPassageQA—was shown to be challenging for both traditional retrieval models and neural architectures. One of the classic approaches to improving retrieval effectiveness across tasks is automatic query expansion (QE). QE is the process of reformulating a user’s query by adding more terms with the goal of retrieving more relevant information. Word embeddings are commonly employed to obtain QE terms by taking advantage of the low dimensional semantic space formed by these embeddings.
Recently, Diaz et al. showed that QE using word embeddings trained on a local query-specific corpus performed better than embeddings that were trained on an entire global corpus for document ranking tasks. We aim to examine the effectiveness of QE, specifically using locally-trained word embeddings, in this new context of answer passage retrieval. Additionally, a query-specific corpus can be small in size with limited vocabulary which forms a challenge for training word embedding models. Since the extent to which limited vocabulary influences the semantic information captured by word embeddings is relatively unexplored, we compare two word embedding models—CBOW and IWE—in this thesis. Having the same underlying training philosophy, the IWE model differs from CBOW in two aspects—it incorporates sub- word information of words and uses a convolutional neural network to learn context representation.
Our results corroborate the findings of Diaz et al.—query-specific data is also beneficial in the task of retrieving passages to open-domain questions. Word embeddings trained on a global corpus fail to capture the nuances of query-specific language present in the answer passages. We also found out that IWE word embeddings capture more semantic information than CBOW word embeddings when trained on local data with a limited vocabulary. Our experiments show that both the IWE model components contribute to the improved quality of word embeddings and consequently better QE terms. Our work can be extended by using the same methodology in other domains or by using different word embedding models to obtain QE terms. The insights from our thesis can help researchers to make an informed decision while choosing word embedding models and training data for their IR and natural language understanding tasks. ... Read more

After surpassing human performance in the fields of Computer Vision, Speech Recognition and NLP, deep learning has been gaining scientific ground in IR. In spite of the sheer amount of publications that have proposed so-called neural IR approaches over the past decade, the field has not achieved the kind of progress seen in related fields. Over the past year or so, works have begun to solve the issues that complicate the progress of neural applications in IR. Among those issues we can find the lack of approaches to interpret and analyze neural IR models, which is addressed in this thesis. We propose a novel approach to diagnose retrieval models that is rooted in the axiomatic approach to IR. Axioms encapsulate search heuristics that are expressed as constraints on retrieval functions. Existing axiomatic approaches have provided fruitful analyses of traditional IR models but are no longer viable to study neural IR models. Building forth on these approaches, we propose a novel approach to empirically analyze retrieval functions, suitable for neural models. Based on inspirations from the NLP and Computer Vision communities, we use model-agnostic diagnostic datasets in order to determine what kind of search heuristics models are able to learn. Since the creation of diagnostic datasets does not require a labeled dataset, we can apply the proposed pipeline to almost any dataset containing queries and documents. We have shown for four specific axioms how to extend and relax them, in order to make them fit for obtaining diagnostic datasets. We have applied our diagnostic dataset creation pipeline to the WikiPassageQA and MSMarco corpora and evaluated three traditional baselines and six neural models. Our experiments on the WikiPassageQA dataset show that the proposed approach can indeed diagnose strengths and weaknesses of neural models. However, our experiments on the MSMarco dataset show that an axiomatic analysis based on the four axioms does not always diagnose factors that incur retrieval effectiveness. An interesting direction for future work is therefore to include more axioms in the diagnostic approach. As possible extensions of the work carried out in this thesis, several roads of future work have been proposed. Among them, we can find reproducing experiments on other neural toolkits and employing the methodology on different IR tasks, but also researching the validity of axioms and adopting a specialized metric for axiomatic performance. We furthermore identified various opportunities to use diagnostic datasets beyond diagnosing neural models. Concluding, we believe that the axiomatic approach to diagnosing neural IR models presented in this work is a step forward to gaining valuable insights into the black boxes that deep models are generally considered to be. We hope our work may prove a fruitful resource for analysis in the field of neural IR on the road towards achieving superior performance without losing sight of a better fundamental understanding of IR. ... Read more

Web search has become a convenient option for seeking information related to learning, therefore understanding how to facilitate human learning through a search engine has the potential to improve the quality of informal education and online learning. One less understood aspect of search as learning is the effect of collaboration in a search session focused on learning. Given the benefits of collaboration during both searching and learning, this thesis aims to evaluate the benefits of collaborative search for a learning-oriented search task.

We have developed a collaborative search system---SearchX---which adopts essential features of collaborative search as found in the literature. In the design process, we focused on providing support for research needs (such as running crowd-sourced experiments and fast prototyping). We deployed SearchX in a crowd-sourced user study on collaborative and single-user search for a learning-oriented search task with a total of 76 participants. Our results show that workspace awareness helped users in understanding the search task at the early stages of the search session and in selecting useful documents. However, we found little evidence that collaboration brought significant changes to the way users search and to users' learning outcomes. We found though that a good rapport between collaborators resulted in higher learning gains, highlighting the importance of social presence for online learning. Finally, we found indications that having a less difficult search topic results in less time spent on understanding search results and more exploration of the topic domain. ... Read more