Previous research on diversity in recommender systems define diversity as the opposite of similarity and propose methods that are based on topic diversity. Diversity in news media, however, is understood as multiperspectivity and scholars generally agree that fostering diversity is the key responsibility of the press in a democratic society. Therefore, a novel viewpoint diversification method was developed, based on the reranking of recommendation lists within the topic using framing aspects. Among other results, an offline evaluation indicated that the proposed method is capable of enhancing the viewpoint diversity of recommendation lists according to a metric from literature. However, to truly enable multiperspectivity in automatic online news environments, users should also be willing to consume viewpoint diverse news recommendation. Therefore, an online study was conducted, assessing how viewpoint diverse recommendations and their presentation characteristics affect the reading behaviour of Blendle users. During a two-week experiment, two groups of 1038 users were presented a set of three recommendations below the content of two articles every day. Thereby, one group received recommendations based on relevance to the original article, while the other group received viewpoint diverse recommendations. Three implicit and one explicit measure of the reading behaviour were analysed. Additionally, the influence of the presentation characteristics of the recommendation on the reading behaviour was analysed. Generally, no major differences were found in the reading behaviour of both user groups. Only the results of the click-through rate calculated per recommendation set indicated a significant difference of 6.5% to the advantage of the baseline users. For the other measures of the reading behaviour, no significant differences were found between the baseline and diverse users. However, the results do show that multiple presentation characteristics have a significant influence on the reading behaviour. Therefore, these results suggest that future research on how recommendation can be presented is just as important as novel viewpoint diversification methods to truly achieve multiperspectivity in automated online news environments.

Visually grounded speech representation learning has shown to be useful in the field of speech representation learning. Studies of learning visually grounded speech embedding adopted speech-image cross-modal retrieval task to evaluate the models, since the cross-modal retrieval task allows to jointly learn both modalities and find their relationships. Specifically, the two modalities, i.e., audio and visual, were jointly embedded into a common space where the speech embeddings and the image embeddings were learned in this process. The obtained embeddings were evaluated by cross-modal retrieval task to see the model performance. Currently, the studies worked on visually grounded speech representation learning trained on scene-based datasets, such as Flickr8k, etc., which learn different objects to infer a new scene. The works that investigating the visually grounded speech representation model's ability to combine different attribute information to infer new objects are lacking. Therefore, this thesis presented a visually grounded speech representation model trained on the fine-grained datasets that contain high level details of objects to learn attribute information associated with objects to infer new objects. The proposed model adopted dual-encoder structure and used different DNN models to extract visual and audio features. An adapted batch loss was used to calculate the similarities between two modalities. Experiments were conducted to test the model performance: 1) The parameter adjusting to obtain a better-performed model. 2) Comparing with state-of-the-art models in speech-image cross-modal retrieval field and fine-grained text-image cross-modal retrieval field. 3) Ablation studies to evaluate components in the model. 4) Research on attention module to see its effectiveness. The results indicated that the proposed model was able to learn the relationships between attributes and objects to retrieve new visual objects and outperformed other visually grounded speech learning models.

This thesis presents a novel self-supervised approach of learning visual representations from videos containing human actions. Our approach tackles the complex problem of learning without the need of labeled data by exploring to what extent the ideas successfully used for images can be transferred, adapted and extended to videos for action recognition purposes. We begin by giving a brief introduction to the topic of learning features without having access to a labeled corpora, providing the motivation of our work. We continue with presenting the related research in terms of contrastive learning, action recognition from videos with 3D convolutions and self-supervised techniques for both images and videos. Next, we formalize our approach with regards to the sampling method, the types of spatial and temporal transformations and the contrastive loss used. We evaluate videoSimCLR proposed method in terms of linear evaluation, fully fine-tuning and video retrieval using two popular action recognition datasets, HMDB51 and UCF101. We also explore the extension of another contrastive learning approach to videos, videoMOCO, and compare it with videoSimCLR by means of linear evaluation.

A Visually Grounded Speech model is a neural model which is trained to embed image caption pairs closely together in a common embedding space. As a result, such a model can retrieve semantically related images given a speech caption and vice versa. The purpose of this research is to investigate whether and how a Visually Grounded Speech model can recognise individual words. Literature on Word Recognition in hu- mans, Automatic Speech Recognition and Visually Grounded Speech models was evaluated. Techniques used to analyse human speech processing, such as gating and priming, were taken as inspiration for the design of the experiments used in this thesis. Multiple aspects of words recognition were investigated through three experiments. Firstly, it was investigated whether the model can recognise individual words. Secondly, it was investigated whether the model can recognise words from a partial sequence of its phonemes. Thirdly, it was investigated how word recognition is affected by contextual information. The experiments show that the model is able to recognise words while not being supervised for that task, and that factors such as word frequency, the length of a word and the speaking rate affect word recognition. Furthermore, the experiments reveal that words can be recognised from a partial input of a word’s phoneme sequence as well, and that recognition is negatively influenced by word competition from the word initial cohort. Furthermore, the word recognition in context experiment reveals that contextual information can enhance the recognition of words which are recognised less well.

Self-Sovereign Identity (SSI) is a new paradigm in digital identity systems that puts the end-user in control: no other actor manages, permits or revokes their digital existence. TrustChain is an academic peer-to-peer networking stack supporting SSI. It delivers passport-grade assurance by integrating with Dutch government. However, end-user control requires a programmed user agent with a human interface and protocols that enable meaningful communication with issuers and verifiers of identity data. This agent must be inter-operable with a large variety of parties and credentials. TrustChain lacks such an interface and protocols.   This thesis makes three main contributions. First, a theoretical framework is proposed for aligning notions of self-sovereignty across contexts, borders and cultures. It provides more detailed, focused and structured discourse than other work and helps consolidate design efforts. Second, a design project is done in collaboration with the Kamer van Koophandel (KVK). It focuses on `authorisation by legal entities', a class of identity problems that have no satisfactory solution yet. Third, a generic common `semantic layer' is prototyped, consisting of a smartphone based user agent and communication protocols. Its wallet-centric approach allows end-users to retrieve their data without leaving the app. The practical value of this prototype is evaluated at a construction site.  The case study shows that the Kamer van Koophandel, like other government institutions, can be a valuable data provider. However, their current legal framework and business model may restrict them. Absence of such vital institutions invites commercial parties to close the gap, threatening privacy and independence of end-users.   Finally, this work has three implications for TrustChain. First, attestation metadata must be considered confidential. Second, single-sided public revocation is required to ensure credential actuality without re-issuing. And third, non-interactive verification enables the construction of chains of untrusted issuers. This is a valuable feature as it enables individuals, not just organisations, to issue claims to others.  

This paper shows the influence of multitasking on the usage of a voice assistant. Voice assistants allow users to input queries over a speech-only channel, and as a result they do not require the same attention as a traditional search engine. Existing research describes the effects of the use of a voice assistant on driving and other demanding activities, however, there is no research that describes how that demanding activity influences the usage of the voice assistant. To research these effects, three sub questions have been constructed and answered. These tackled three characteristics to describe the usage: the query formulation, the knowledge gain and the user experience. To answer these questions a user study was conducted where the participants used a voice assistant in three situations. Two of these situations included a distraction in the form of a game. We found that the presence of dual tasking results in shorter queries. We also found that a higher intensity of the session can decrease the knowledge gain.

This paper delved into the effects of domain expertise on a user's conversational search, because as the use and acceptance of voice assistants increase the need for conversational search agent that can accommodate to a human characteristic such as domain expertise. Accommodating to disadvantaged users of web search as earlier works showed that users with low literacy and low spatial visualization abilities are strongly affected in their searches compared to users who do not suffer from these impairments. Prior research into domain expertise demonstrated the influence it has had on the querying behavior of users in web search. They found that domain experts included more domain specific jargon in their messages, made longer queries and spent less time per search task. This paper examined these findings in a conversational search setting. Contrary to these findings, no significant relation between the domain expertise level and any of these results could be established. However, conducting the experiment to assess these findings has provided insight into how users respond to a conversational search study such as this one.

Recommender systems focus on automatically surfacing suitable items for users from digital collections that are too large for the user to oversee themselves. A considerable body of work exists on surfacing items that match what a user liked in the past; this way, the recommender system will exploit its knowledge of a user's comfort zone. However, application scenarios exist in which it is explicitly important to offer the user opportunities to explore items beyond their existing comfort zones. In such cases, the recommender should include items for which there is less existing evidence that the user will like them. This calls for the recommender to explore to what extent a user would be tolerant to exploration. In this thesis, we consider that different users will likely have different preferences with respect to item exploration. We propose personalized item filtering techniques for this, modeled under a multi-armed bandit framework, that consider (1) how and where exploration vs. exploitation items should be distributed in a result overview and (2) how adventurous exploration items can be, considering a user's general willingness to explore and existing familiarity with item categories in the collection. We present the results of a survey and an online quantitative experiment with 43 users of Muziekweb, a public music library that encourages taste broadening in the Netherlands, demonstrating the effectiveness of both our proposed filtering techniques that aim for personalized exploration.

With the increasing popularity of mobile and voice-assisted, extracting short and precise answer passages to open-domain questions is becoming an increasingly important information retrieval (IR) task. The recently released large-scale corpus for answer passage retrieval—WikiPassageQA—was shown to be challenging for both traditional retrieval models and neural architectures. One of the classic approaches to improving retrieval effectiveness across tasks is automatic query expansion (QE). QE is the process of reformulating a user’s query by adding more terms with the goal of retrieving more relevant information. Word embeddings are commonly employed to obtain QE terms by taking advantage of the low dimensional semantic space formed by these embeddings.
Recently, Diaz et al. showed that QE using word embeddings trained on a local query-specific corpus performed better than embeddings that were trained on an entire global corpus for document ranking tasks. We aim to examine the effectiveness of QE, specifically using locally-trained word embeddings, in this new context of answer passage retrieval. Additionally, a query-specific corpus can be small in size with limited vocabulary which forms a challenge for training word embedding models. Since the extent to which limited vocabulary influences the semantic information captured by word embeddings is relatively unexplored, we compare two word embedding models—CBOW and IWE—in this thesis. Having the same underlying training philosophy, the IWE model differs from CBOW in two aspects—it incorporates sub- word information of words and uses a convolutional neural network to learn context representation.
Our results corroborate the findings of Diaz et al.—query-specific data is also beneficial in the task of retrieving passages to open-domain questions. Word embeddings trained on a global corpus fail to capture the nuances of query-specific language present in the answer passages. We also found out that IWE word embeddings capture more semantic information than CBOW word embeddings when trained on local data with a limited vocabulary. Our experiments show that both the IWE model components contribute to the improved quality of word embeddings and consequently better QE terms. Our work can be extended by using the same methodology in other domains or by using different word embedding models to obtain QE terms. The insights from our thesis can help researchers to make an informed decision while choosing word embedding models and training data for their IR and natural language understanding tasks.

With the advent of social robots which are designed to be 'social', human-like interactions have become a necessity. It is natural for us to use a plethora of emotions to convey additional information or to make an interaction more engaging. But emotion expression is not commonly associated with robots. Many humanoid robots cannot generate facial expressions to portray various emotions. Studies have shown that robots are multi-modal systems which can employ multiple channels to express an emotion. This thesis explores the non-verbal emotion expression techniques and their expressive capabilities. We found that some emotions are easier to express than others, and a single technique cannot express all the emotions. We chose a few emotions and systematically determined the best technique for each of them.

This thesis focuses on the field of Job Recommendation. Particularly, we focus on using implicit preferences exhibited by the job seeker in interactions with a web platform to propose an improved ranking algorithm for a job recommendation platform called Magnet.me. We also study evaluation of relevance, and evaluation of recommendation sorting algorithms to determine the degree of improvement achieved by the proposed algorithm. Using NDCG with different relevance evaluations, we test performance of the proposed algorithm in an online experiment on the job recommendation platform. We find that the evaluation of relevance strongly affects the distinguishability of NDCG. The evaluation shows that our sorting algorthm outperforms the original algorithm when using classical binary relevance, or relevance evaluations that consider items with negative feedback less relevant than items with missing feedback. However, when using relevance evaluations for NDCG that punish missing feedback more than negative feedback, NDCG loses its capability of distinguishing between algorithm performance. Based on baseline sorting algorithm evaluation MRR and the different evaluations using NDCG, we conclude that the proposed recommendation sorting algorithm outperforms the original algorithm.

There is an increasing attention towards link prediction in complex networks both in physical and computer science communities. Particularly Online Social Networks (OSNs) are
becoming the most popular platforms for information sharing, content creation and communication between users on the Internet. However, most of the research was done considering only a static snapshot of the network and without using relevant information from
other types of activities.
In that direction, the present thesis proposes a novel method for link prediction using temporal information in Stack Overflow with the assistance of interactions from Github. The
developed multilayer network enhanced with temporal interactions is aiming to improve
the performance of the prediction compared to the traditional methods while the design
choices intend to investigate the evolution of the network through time. In the end, the
generalized framework could be used not only to make accurate link prediction that translate to human interactions over time, but also as a tool to characterize the behavior of the
users in the targeted network.

Nowadays, most users access the web through search engine portals. However, information needs can often be ill-defined or too broad to be solvable by a list of results the user has to scroll through, which implies that he is most likely required to refine the need by himself to reach the desired result. In recent years, researchers have attempted to tackle these issues through conversations, more specifically through conversational search. This topic has seen an increase of interest from the research community, proven by the appearance of specialized workshops and seminars. The general public has also started to show interest, proven by the emergence of a wide range of virtual assistants, such as Google Assistant, Microsoft Cortana or Amazon Alexa. As such conversational systems seek to fulfill an information need of a user, they should be able to elicit and fully understand his requirements regardless of the domain, track the conversation as it evolves while attempting to clarify the initial information need and provide suggestions and answers that are based on concrete knowledge sources. Although various developments in domains adjacent to conversational search enabled us to better understand natural language, there is a lack of large-scale datasets that are appropriate for training models to perform conversational search tasks. Through our research, we have built a collection of over 80,000 conversations that fulfill the requirements of a conversational search dataset. We have benchmarked this dataset on three distinct tasks using multiple baselines.

Embodied Conversational Agents (ECA) seek to provide a more natural means of interaction for a user through verbal and non-verbal properties of human face-to-face communication. For this reason, these systems are found to bring benefits in different mental health related interventions. However, a key challenge in developing agents to replace the human interlocutor in a dyadic conversation, is to simulate appropriate attentive listening behaviors. In this thesis work, we explored different backchannel strategies and studied their effects in terms of likability and engagement. We built a fully embodied conversational agent with three different levels of backchannel strategies and ran a within-subject study with a convenience sample of 24 participants. The results showed that the amount of emotional words in the speech of users increased if the attentive listening capabilities of the agent were improved. In addition, the capability to trigger both verbal and nonverbal backchannels with proper timing was found to be a relevant feature in terms of improved speech rate and emotional words. Contrary to our hypothesis, backchannels based on actual emotion and sentiment analysis of the speech content were not found to be significantly influential on the quality of interaction. Multi-modal approaches are suggested for future works in order to overcome limitations of this work due to potential lack in emotion detection accuracy.

This research is aimed at breaking online political filter bubbles. We present a system that uses artificial intelligence and human computation to automatically collect article metadata on political content and thereby enables diverse personalization on content-serving platforms. This way, exploring alternative viewpoints could become as simple as checking your feed.

This research is aimed at breaking online political filter bubbles. We present a system that uses artificial intelligence and human computation to automatically collect article metadata on political content and thereby enables diverse personalization on content-serving platforms. This way, exploring alternative viewpoints could become as simple as checking your feed.

LeQuest develops interactive e-training modules to improve the competence regarding medical technology of medical professionals. The medical technology is analysed by LeQuest to develop training modules, but the analysis process and writing associated information can be performed more efficiently. This would reduce the required time and resources which could be invested in additional trainings and quality improvements. In the end, this will lead to an improvement regarding the patient's safety in health institutions. This work empirically evaluated the Spoofax Workbench by conducting an industrial case-study which consists of the design, implementation and evaluation of a domain-specific language (DSL). The LeQuest DSL is used as a tool for transforming the current analysis process into a more formalized process which does allow for objective observations, measurements and quantifiable information. Although the LeQuest DSL is not integrated in the current work-flow yet, the evaluation has shown that it is expected that the overall quality and efficiency of the analysis process will increase after the introduction of the DSL.