A Visually Grounded Speech model is a neural model which is trained to embed image caption pairs closely together in a common embedding space. As a result, such a model can retrieve semantically related images given a speech caption and vice versa. The purpose of this research is to investigate whether and how a Visually Grounded Speech model can recognise individual words. Literature on Word Recognition in hu- mans, Automatic Speech Recognition and Visually Grounded Speech models was evaluated. Techniques used to analyse human speech processing, such as gating and priming, were taken as inspiration for the design of the experiments used in this thesis. Multiple aspects of words recognition were investigated through three experiments. Firstly, it was investigated whether the model can recognise individual words. Secondly, it was investigated whether the model can recognise words from a partial sequence of its phonemes. Thirdly, it was investigated how word recognition is affected by contextual information. The experiments show that the model is able to recognise words while not being supervised for that task, and that factors such as word frequency, the length of a word and the speaking rate affect word recognition. Furthermore, the experiments reveal that words can be recognised from a partial input of a word’s phoneme sequence as well, and that recognition is negatively influenced by word competition from the word initial cohort. Furthermore, the word recognition in context experiment reveals that contextual information can enhance the recognition of words which are recognised less well.

A Visually Grounded Speech model is a neural model which is trained to embed image caption pairs closely together in a common embedding space. As a result, such a model can retrieve semantically related images given a speech caption and vice versa. The purpose of this research is to investigate whether and how a Visually Grounded Speech model can recognise individual words. Literature on Word Recognition in hu- mans, Automatic Speech Recognition and Visually Grounded Speech models was evaluated. Techniques used to analyse human speech processing, such as gating and priming, were taken as inspiration for the design of the experiments used in this thesis. Multiple aspects of words recognition were investigated through three experiments. Firstly, it was investigated whether the model can recognise individual words. Secondly, it was investigated whether the model can recognise words from a partial sequence of its phonemes. Thirdly, it was investigated how word recognition is affected by contextual information. The experiments show that the model is able to recognise words while not being supervised for that task, and that factors such as word frequency, the length of a word and the speaking rate affect word recognition. Furthermore, the experiments reveal that words can be recognised from a partial input of a word’s phoneme sequence as well, and that recognition is negatively influenced by word competition from the word initial cohort. Furthermore, the word recognition in context experiment reveals that contextual information can enhance the recognition of words which are recognised less well.

Previous research on diversity in recommender systems define diversity as the opposite of similarity and propose methods that are based on topic diversity. Diversity in news media, however, is understood as multiperspectivity and scholars generally agree that fostering diversity is the key responsibility of the press in a democratic society. Therefore, a novel viewpoint diversification method was developed, based on the reranking of recommendation lists within the topic using framing aspects. Among other results, an offline evaluation indicated that the proposed method is capable of enhancing the viewpoint diversity of recommendation lists according to a metric from literature. However, to truly enable multiperspectivity in automatic online news environments, users should also be willing to consume viewpoint diverse news recommendation. Therefore, an online study was conducted, assessing how viewpoint diverse recommendations and their presentation characteristics affect the reading behaviour of Blendle users. During a two-week experiment, two groups of 1038 users were presented a set of three recommendations below the content of two articles every day. Thereby, one group received recommendations based on relevance to the original article, while the other group received viewpoint diverse recommendations. Three implicit and one explicit measure of the reading behaviour were analysed. Additionally, the influence of the presentation characteristics of the recommendation on the reading behaviour was analysed. Generally, no major differences were found in the reading behaviour of both user groups. Only the results of the click-through rate calculated per recommendation set indicated a significant difference of 6.5% to the advantage of the baseline users. For the other measures of the reading behaviour, no significant differences were found between the baseline and diverse users. However, the results do show that multiple presentation characteristics have a significant influence on the reading behaviour. Therefore, these results suggest that future research on how recommendation can be presented is just as important as novel viewpoint diversification methods to truly achieve multiperspectivity in automated online news environments.

Previous research on diversity in recommender systems define diversity as the opposite of similarity and propose methods that are based on topic diversity. Diversity in news media, however, is understood as multiperspectivity and scholars generally agree that fostering diversity is the key responsibility of the press in a democratic society. Therefore, a novel viewpoint diversification method was developed, based on the reranking of recommendation lists within the topic using framing aspects. Among other results, an offline evaluation indicated that the proposed method is capable of enhancing the viewpoint diversity of recommendation lists according to a metric from literature. However, to truly enable multiperspectivity in automatic online news environments, users should also be willing to consume viewpoint diverse news recommendation. Therefore, an online study was conducted, assessing how viewpoint diverse recommendations and their presentation characteristics affect the reading behaviour of Blendle users. During a two-week experiment, two groups of 1038 users were presented a set of three recommendations below the content of two articles every day. Thereby, one group received recommendations based on relevance to the original article, while the other group received viewpoint diverse recommendations. Three implicit and one explicit measure of the reading behaviour were analysed. Additionally, the influence of the presentation characteristics of the recommendation on the reading behaviour was analysed. Generally, no major differences were found in the reading behaviour of both user groups. Only the results of the click-through rate calculated per recommendation set indicated a significant difference of 6.5% to the advantage of the baseline users. For the other measures of the reading behaviour, no significant differences were found between the baseline and diverse users. However, the results do show that multiple presentation characteristics have a significant influence on the reading behaviour. Therefore, these results suggest that future research on how recommendation can be presented is just as important as novel viewpoint diversification methods to truly achieve multiperspectivity in automated online news environments.

Since the introduction of the Web, online platforms have become a place to share opinions across various domains (e.g., social media platforms, discussion fora or webshops). Consequently, many researchers have seen a need to classify, summarise or categorise these large sets of unstructured user-generated content. A field related to this task is also known as opinion mining in which various applications have focused on sentiment analysis techniques to classify opinionated documents based on sentiment. More recent, researchers have focused on stance classification to classify opinionated documents based on stance in controversial debates. However, in the case of such controversial debates it would be equally interesting to know the underlying reasons behind a stance in order to truly understand a discussion. We can call these underlying reasons as perspectives. Few have focused on distilling such perspectives from text and in this research we aim to explore the use of an unsupervised model - called joint topic models - to perform the task of perspective discovery. We define perspective discovery on a controversial debate as the process of automatically finding and extracting a structured overview of perspectives from unstructured text. The aim is to quantify how well existing joint topic models can extract human understandable perspectives between and within stances for more fine-grained opinion mining on textual debates. To perform this evaluation we propose an evaluation setup with an extensive user study. This setup focuses on the topic model’s clustering ability of perspectives as well as the human understandability of the topic model’s output. Based on the results we may derive that topic models can discover some of the perspectives from text. Moreover, the results suggest that users are not influenced by their pre-existing stance when interpreting the output of topic models.

Since the introduction of the Web, online platforms have become a place to share opinions across various domains (e.g., social media platforms, discussion fora or webshops). Consequently, many researchers have seen a need to classify, summarise or categorise these large sets of unstructured user-generated content. A field related to this task is also known as opinion mining in which various applications have focused on sentiment analysis techniques to classify opinionated documents based on sentiment. More recent, researchers have focused on stance classification to classify opinionated documents based on stance in controversial debates. However, in the case of such controversial debates it would be equally interesting to know the underlying reasons behind a stance in order to truly understand a discussion. We can call these underlying reasons as perspectives. Few have focused on distilling such perspectives from text and in this research we aim to explore the use of an unsupervised model - called joint topic models - to perform the task of perspective discovery. We define perspective discovery on a controversial debate as the process of automatically finding and extracting a structured overview of perspectives from unstructured text. The aim is to quantify how well existing joint topic models can extract human understandable perspectives between and within stances for more fine-grained opinion mining on textual debates. To perform this evaluation we propose an evaluation setup with an extensive user study. This setup focuses on the topic model’s clustering ability of perspectives as well as the human understandability of the topic model’s output. Based on the results we may derive that topic models can discover some of the perspectives from text. Moreover, the results suggest that users are not influenced by their pre-existing stance when interpreting the output of topic models.

In this thesis, we study automatically generating explanatory reports for anomalous incidents in a train control system (TCS) using Natural Language Generation (NLG). A TCS is a type of safety-critical software that allows train controllers to correctly set the tracks for a train to pass. The goal of this research is to process the majority of log files generated by such a system, detect anomalies that have occurred and represent that data in human-readable reports that explain anomalous incidents. The reports are generated by making use of NLG techniques, namely data-to-text. We design an NLG pipeline that incorporates novel graphical components. We perform all steps of the report generation process, which is the processing of data, anomaly detection and representing the data in natural language. To process the data we incorporate complex domain-specific rules which require extensive work to accommodate five separate types of event log files. Analyzing logs and detecting anomalous incidents proved complex due to the complicated structure of logs and the ill-defined relationships between different log file types. The log data explained with the NLG pipeline is more extensive than we have seen in literature, as it needs to be linked using complex domain-specific rules and includes temporal and geographical aspects of the railway system. Furthermore, the detected anomalies are very diverse and thus it is unclear what aspects should be included in a report for each anomalous case. We research whether a purely textual presentation or a combination of modalities provides higher information quality. The product of the NLG system are three presentation versions of incident reports, Text, TimeText and RailText. Each version focuses on different important aspects of the nature of the data, to explain anomalous incidents. Due to the diverse properties of the anomalous cases detected, we implement a Case-Based Reasoning (CBR) system to predict the appropriate presentation to explain each incident. We evaluate our work in two expert-based evaluation phases. The first phase evaluates the quality of the different report presentations. The second phase evaluates the feasibility of the CBR system and the quality of the reports chosen by CBR. The work in this thesis is significant as we manage to translate complex data into human-readable reports that are well received by experts and help in understanding anomalous cases. We learn that there is a difference in what presentation format provides the highest information quality of reports depending on the anomalous case being explained. Therefore we conclude that the properties of anomalous cases should be taken into account when generating anomalous reports in a TCS to ensure the highest perceived information quality of the reports. CBR is shown to perform that task well. Furthermore, we find that data familiarity of experts effects their preferences for report presentations.

In this thesis, we study automatically generating explanatory reports for anomalous incidents in a train control system (TCS) using Natural Language Generation (NLG). A TCS is a type of safety-critical software that allows train controllers to correctly set the tracks for a train to pass. The goal of this research is to process the majority of log files generated by such a system, detect anomalies that have occurred and represent that data in human-readable reports that explain anomalous incidents. The reports are generated by making use of NLG techniques, namely data-to-text. We design an NLG pipeline that incorporates novel graphical components. We perform all steps of the report generation process, which is the processing of data, anomaly detection and representing the data in natural language. To process the data we incorporate complex domain-specific rules which require extensive work to accommodate five separate types of event log files. Analyzing logs and detecting anomalous incidents proved complex due to the complicated structure of logs and the ill-defined relationships between different log file types. The log data explained with the NLG pipeline is more extensive than we have seen in literature, as it needs to be linked using complex domain-specific rules and includes temporal and geographical aspects of the railway system. Furthermore, the detected anomalies are very diverse and thus it is unclear what aspects should be included in a report for each anomalous case. We research whether a purely textual presentation or a combination of modalities provides higher information quality. The product of the NLG system are three presentation versions of incident reports, Text, TimeText and RailText. Each version focuses on different important aspects of the nature of the data, to explain anomalous incidents. Due to the diverse properties of the anomalous cases detected, we implement a Case-Based Reasoning (CBR) system to predict the appropriate presentation to explain each incident. We evaluate our work in two expert-based evaluation phases. The first phase evaluates the quality of the different report presentations. The second phase evaluates the feasibility of the CBR system and the quality of the reports chosen by CBR. The work in this thesis is significant as we manage to translate complex data into human-readable reports that are well received by experts and help in understanding anomalous cases. We learn that there is a difference in what presentation format provides the highest information quality of reports depending on the anomalous case being explained. Therefore we conclude that the properties of anomalous cases should be taken into account when generating anomalous reports in a TCS to ensure the highest perceived information quality of the reports. CBR is shown to perform that task well. Furthermore, we find that data familiarity of experts effects their preferences for report presentations.

News is the main source of information about events in our neighborhood and around the globe. In an era of digital news, where sources vary in quality and news spreads fast, it is critical to understand what is being consumed by the public. Partisan news is news that favors certain political parties or ideologies. It is undesirable because news organization should aim for an objective and balanced reporting. An automatic system that can classify news to be partisan or non-partisan is thus desired. Such a system (partisan news detector) requires a sufficient amount of data to learn the pattern of partisanship in news articles. However, these labels are expensive to collect since manual inspection of each article is required.

Inferring partisanship labels from the partisanship of publishers is another approach that has been used in previous research. By treating all articles by partisan publishers to be partisan news and those by non-partisan publishers to be non-partisan, it is easy to collect a large number of labeled articles. This way of deciding labels is noisy, making it more difficult for a detector to learn.

This thesis compared the performance of using publisher-level labels and article-level labels for partisan news detection. The detector was designed as a binary classifier. We compared the difference in performance across several feature sets to ensure the observed difference was due to the annotation level, not the choice of specific classifiers. The experiments were performed on two datasets of different properties to ensure the generalizability of the results. We found that classifiers trained with publisher-level labels have higher recall but lower F1-score compared to classifiers trained with article-level labels. We also observed that the classifiers overfit on publishers but reducing the overfitting with feature selection did not improve the performance. Comparing the performance difference between the two datasets, we concluded that an important factor that determines the performance achievable by the publisher-level labels is the quality of publishers that are included in the dataset. This is valuable for future dataset collection.

Our work provides a benchmark performance of publisher-level labels, which can be used as baselines for future research that investigate other methodologies to utilize the noisy labels. We also compared the performance between the two levels of annotation and concluded that partisan news detectors trained with article-level labels are more practical to be used in a fully-automated system since they have on average 10\% higher F1-scores than those trained with publisher-level labels. However, the high recall of the latter makes them applicable in use cases where high recall is desired.

News is the main source of information about events in our neighborhood and around the globe. In an era of digital news, where sources vary in quality and news spreads fast, it is critical to understand what is being consumed by the public. Partisan news is news that favors certain political parties or ideologies. It is undesirable because news organization should aim for an objective and balanced reporting. An automatic system that can classify news to be partisan or non-partisan is thus desired. Such a system (partisan news detector) requires a sufficient amount of data to learn the pattern of partisanship in news articles. However, these labels are expensive to collect since manual inspection of each article is required.

Inferring partisanship labels from the partisanship of publishers is another approach that has been used in previous research. By treating all articles by partisan publishers to be partisan news and those by non-partisan publishers to be non-partisan, it is easy to collect a large number of labeled articles. This way of deciding labels is noisy, making it more difficult for a detector to learn.

This thesis compared the performance of using publisher-level labels and article-level labels for partisan news detection. The detector was designed as a binary classifier. We compared the difference in performance across several feature sets to ensure the observed difference was due to the annotation level, not the choice of specific classifiers. The experiments were performed on two datasets of different properties to ensure the generalizability of the results. We found that classifiers trained with publisher-level labels have higher recall but lower F1-score compared to classifiers trained with article-level labels. We also observed that the classifiers overfit on publishers but reducing the overfitting with feature selection did not improve the performance. Comparing the performance difference between the two datasets, we concluded that an important factor that determines the performance achievable by the publisher-level labels is the quality of publishers that are included in the dataset. This is valuable for future dataset collection.

Our work provides a benchmark performance of publisher-level labels, which can be used as baselines for future research that investigate other methodologies to utilize the noisy labels. We also compared the performance between the two levels of annotation and concluded that partisan news detectors trained with article-level labels are more practical to be used in a fully-automated system since they have on average 10\% higher F1-scores than those trained with publisher-level labels. However, the high recall of the latter makes them applicable in use cases where high recall is desired.

With the advent of social robots which are designed to be 'social', human-like interactions have become a necessity. It is natural for us to use a plethora of emotions to convey additional information or to make an interaction more engaging. But emotion expression is not commonly associated with robots. Many humanoid robots cannot generate facial expressions to portray various emotions. Studies have shown that robots are multi-modal systems which can employ multiple channels to express an emotion. This thesis explores the non-verbal emotion expression techniques and their expressive capabilities. We found that some emotions are easier to express than others, and a single technique cannot express all the emotions. We chose a few emotions and systematically determined the best technique for each of them.

With the advent of social robots which are designed to be 'social', human-like interactions have become a necessity. It is natural for us to use a plethora of emotions to convey additional information or to make an interaction more engaging. But emotion expression is not commonly associated with robots. Many humanoid robots cannot generate facial expressions to portray various emotions. Studies have shown that robots are multi-modal systems which can employ multiple channels to express an emotion. This thesis explores the non-verbal emotion expression techniques and their expressive capabilities. We found that some emotions are easier to express than others, and a single technique cannot express all the emotions. We chose a few emotions and systematically determined the best technique for each of them.

Consumers share their experiences or opinion about products or brands in various channels nowadays, for example on review websites or social media. Sentiment analysis is used to predict the sentiment of text from consumers about these products or brands in order to understand the tone of customers towards these products or brands. This thesis addresses sentiment analysis in the product domain on sentence level. In this thesis three data types are used which are collected by Unilever, review data which is text that contains the opinion of a customer towards a specific product. Social data, which can be tweets, Facebook messages, Instagram messages etc. and phone data which is a summary of a phone call of a customer about a specific product.

When conducting sentiment analysis one solution is to extract features from the data which can be given to a machine learning algorithm together with sentiment labels given by human annotators. The machine learning algorithm will generate a classifier which can predict a label for sentences.
In sentiment analysis literature it is often not clear why certain features are chosen or for which data type certain features will work well. In this research we compare the differences when using several feature sets for the different data types.

We propose three feature sets for review data and three feature sets for social data. We focus on two aspects, comparing the different feature sets and comparing the data types. In our results we do not find significant differences in performance between the feature sets. The results suggest there might be feature sets which can improve sentiment analysis specifically for the data type, but a general feature set with standard features can be comparable to that result.

Consumers share their experiences or opinion about products or brands in various channels nowadays, for example on review websites or social media. Sentiment analysis is used to predict the sentiment of text from consumers about these products or brands in order to understand the tone of customers towards these products or brands. This thesis addresses sentiment analysis in the product domain on sentence level. In this thesis three data types are used which are collected by Unilever, review data which is text that contains the opinion of a customer towards a specific product. Social data, which can be tweets, Facebook messages, Instagram messages etc. and phone data which is a summary of a phone call of a customer about a specific product.

When conducting sentiment analysis one solution is to extract features from the data which can be given to a machine learning algorithm together with sentiment labels given by human annotators. The machine learning algorithm will generate a classifier which can predict a label for sentences.
In sentiment analysis literature it is often not clear why certain features are chosen or for which data type certain features will work well. In this research we compare the differences when using several feature sets for the different data types.

We propose three feature sets for review data and three feature sets for social data. We focus on two aspects, comparing the different feature sets and comparing the data types. In our results we do not find significant differences in performance between the feature sets. The results suggest there might be feature sets which can improve sentiment analysis specifically for the data type, but a general feature set with standard features can be comparable to that result.