On Fine-grained Temporal Emotion Recognition in Video: How to Trade off Recognition Accuracy with Annotation Complexity?

Fine-grained emotion recognition is the process of automatically identifying the emotions of users at a fine granularity level, typically in the time intervals of 0.5s to 4s according to the expected duration of emotions. Previous work mainly focused on developing algorithms to recognize only one emotion for a video based on the user feedback...

Increasing trust in complex machine learning systems: Studies in the music domain

Machine learning (ML) has become a core technology for many real-world applications. Modern ML models are applied to unprecedentedly complex and difficult challenges, including very large and subjective problems. For instance, applications towards multimedia understanding have been advanced substantially. Here, it is already prevalent that...

Information Diffusion on Temporal Networks

As an important carrier of information diffusion, social media has experienced a huge increase in the number of users and also has a big effect on the way of how information diffuses. For example, Facebook and Youtube have attracted more than 1.6 and 1.3 billion users until 2020, respectively. The use of internet and online social network have...

On predicting individual video viewing experience: The value of user information

Experience prediction is one key component in today’s multimedia delivery. Knowing user’s viewing experience allows online video service providers (e.g., Netflix, YouTube) to create value for their customers by providing personalized content and service. However, individual experience prediction is a challenging problem since viewing experience ...

Music in Use: Novel perspectives on content-based music Retrieval

Music consumption has skyrocketed in the past few years with advancements in internet and streaming technologies. This has resulted in the rapid development of the inter-disciplinary field of Music Information Retrieval (MIR), which develops automatic methods to efficiently and effectively access the wealth of musical content. In general,...

Advanced Factorization Models for Recommender Systems

Recommender Systems have become a crucial tool to serve personalized content and to promote online products and media, but also to recommend restaurants, events, news and dating profiles. The underlying algorithms have a significant impact on the quality of recommendations and have been the subject of many studies in the last two decades. In...

Visual Quality of Experience: A Metric-Driven Perspective

Multimedia systems are typically optimized in a way that maximizes users’ satisfaction of using the systems/services. This user satisfaction is what is commonly referred to as Quality of Experience (QoE). For visual media, such as images and videos, the optimization of QoE has meant reducing the visibility of artifacts (e.g. noise or other...

Music Information Retrieval beyond Audio: A Vision-based Approach for Real-world Data

Digital music platforms have recently become the primary revenue stream for recorded music, making record labels and content owners increasingly interested in developing new digital features for their users.<br/>Besides listening to expert-curated playlists and automatically recommended music, users can also benefit from a more informative, non...

Incorporating Crowd Perspectives into Multimedia Retrieval Systems

The twenty-first century has brought plentiful computational power and bandwidth to the masses and has opened up access to multimedia recording devices for everyone. With these developments, a shift in the landscape of multimedia took place: from traditional one-to-many programming (the paradigm of traditional television) to many-to-many...

Dynamic processes on complex networks: The role of heterogeneity

In the recent decades, various dynamic process models on complex networks have been built to study the mechanisms by which an opinion, a disease or generally the information spreads in real-world networks. For example, opinion models are developed to illustrate the competition of opinions in a population, and epidemic models are used to describe...

Large Scale Image Retrieval for Location Estimation

The geo-graphical location at which an image or video was taken is a key piece of multimedia information. Such geo-information has become an indispensable component of systems enabling personalized and context-aware multimedia services. The research reported in this thesis investigates how to automatically derive geo-information from multimedia...

Multifaceted Approaches to Music Information Retrieval

Music is a multifaceted phenomenon: beyond addressing our auditory channel, the consumption of music triggers further senses. Also in creating and communicating music, multiple modalities are at play. Next to this, it allows for various ways of interpretation: the same musical piece can be performed in different valid ways, and audiences can in...

User Intent in Online Video Search

Over the recent years, user expectations of the ability of video search engines have significantly risen. Users expect video search engines to be useful as an instrument that facilitates communication, education, entertainment and problem solving and, in relation to this, to satisfy diverse information needs. A user's information need is the...

Advanced Retrieval Models for Web Image Search

Ranking and Context-awareness in Recommender Systems

In this thesis we report the results of our research on recommender systems, which addresses some of the critical scientific challenges that still remain open in this domain. Collaborative filtering (CF) is the most common technique of predicting the interests of a user by collecting preference information from many users. In order to determine...

Advancing the Relevance Criteria for Video Search and Visual Summarization

To facilitate finding of relevant information in ever-growing multimedia collections, a number of multimedia information retrieval solutions have been proposed over the past years. The essential element of any such solution is the relevance criterion deployed to select or rank the items from a multimedia collection to be presented to the user....

Content Discovery from Composite Audio: An unsupervised approach

In this thesis, we developed and assessed a novel robust and unsupervised framework for semantic inference from composite audio signals. We focused on the problem of detecting audio scenes and grouping them into meaningful clusters. Our approach addressed all major steps in a general process of composite audio analysis, from low-level signal...

Random sampling methods for two-view geometry estimation

This thesis treats efficient estimation algorithms for the epipolar geometry, the model underlying two views of the same scene or object. The epipolar geometry is computed from image correspondences that are found by local feature matching. These correspondences are used to calculate the fundamental matrix, which is the mathematical...