Spreading processes are ubiquitous in nature and society, from the diffusion of information in social platforms to the spread of diseases within populations. Many real-world systems can be represented as networks, where a piece of information or a disease spreads along links connecting nodes. Different nodes and links often differ in their network properties and play distinct roles in a spreading process. Based on network properties of nodes or links, practitioners may be interested in identifying key nodes as the seed nodes to maximally diffuse a piece of information, or removing specific links to mitigate the spreading. In this thesis, we study the roles of a node or a link in a spreading process from three different perspectives and investigate how these roles relate to the properties of nodes and links within the underlying network.

We first explore how the network properties of a node can be used to predict the spreading influence of the node, defined as the average number of nodes that are ultimately infected when this node is the only seed node. Previous studies have shown that combining node properties derived from local and global topological information can better predict nodal influence than using a single metric. In Chapter 2, we investigate whether using relatively local information is sufficient for the prediction. To address this question, we define an iterative metric set by leveraging the iterative process used to derive classical nodal centralities like eigenvector centrality. The iterative metric set progressively incorporates more global information and is used as the feature set in a regression model to predict nodal spreading influence. The iterative metric set is then used as the feature set in a regression model to predict the spreading influence of a node. We find that the model using the iterative metric set that includes relatively local information achieves comparable prediction quality with the method that includes both local and global information, in various networks.

A spreading process can be mitigated by blocking social contacts, i.e., time-specific interactions. In Chapter 3, we investigate how the network properties of a contact are associated with the mitigation effect when the contact is blocked. We develop probabilistic contact blocking strategies, which remove contacts (temporal links) based on their properties in a temporal network, to mitigate the spread of a Susceptible-Infected-Recovered spreading process. The removal probability of a contact depends on a given centrality metric of the corresponding link in the time-aggregated network and the occurring time of the contact. We propose diverse link centrality metrics, and each centrality metric leads to a unique contact blocking strategy. Our results indicate that the spread of the epidemic is most effectively mitigated when contacts between node pairs that have fewer contacts and contacts that occur earlier in time are more likely to be removed.

The role of a link in a spreading process can also be reflected by the extent to which the link is used in the process. Many real-world systems may involve interactions among groups of more than two individuals and can therefore be represented as temporal higher-order networks. Chapter 4 explores the Susceptible-Infected threshold spreading process unfolding on temporal higher-order networks with two objectives: (1) to understand the contribution of each hyperlink to the spreading process, defined as the average number of nodes that are directly infected via the activation of the hyperlink starting from an arbitrary seed node, and (2) to investigate hyperlinks with what network properties tend to contribute more to the spreading process. This understanding is crucial for developing effective strategies to mitigate a spreading process. Given a temporal higher-order network, we propose to construct a weighted higher-order network, the so-called diffusion backbone, where the weight of each hyperlink denotes its contribution to the spreading process. We then systematically design centrality metrics for hyperlinks in a temporal higher-order network, where each centrality metric captures a specific property of the hyperlink within a temporal higher-order network and is used to estimate the ranking of hyperlinks by their weights in the backbone. We find and explain why certain centrality metrics can better estimate the contributions of hyperlinks under different parameters of the spreading process.

The last chapter reflects on the insights of this thesis and discusses possible future directions related to our research.

Extending the concepts of classical signal processing to graphs, a wide array of methods have come to the fore, including filtering, reconstruction, classification, and sampling. Existing approaches in graph signal processing consider a known and static topology, i.e., fixed number of nodes and a fixed edge support. Two types of tasks stand out, namely, topology inference, where the edge support along with their weights are estimated from signals; and data processing, where existing data and the known topology are used to perform different tasks. However, such tasks become quite challenging when the network size and support changes over time. Particularly, these challenges involve adapting to the changing topology, data distributions and dealing with unknown topological information. The latter manifests for example, when new nodes are available to attach to the graph but their connectivity is uncertain as is the case in cold start graph-based recommender systems.
The key contribution of this dissertation is proposing methodologies for signal processing over dynamic networks which are aimed at the two aforementioned tasks. For dynamic networks with incoming nodes, we process signals by introducing a parametric stochastic attachment model. In this model, the incoming nodes connect with probability to existing nodes with certain weights. This uncertainty allows us to model input output relations and allows us to cast them in the context of different graph signal processing tasks. We learn the model attachment parameters in a task-aware setting, allowing us to interpret topology identification in task-aware settings. Separately, we also propose filter design strategies for processing signals both at the incoming and existing nodes using stochastic attachment models.
Another contribution of this dissertation is to extend graph signal processing with graph filters to the scenario where the graph keeps growing in size with streaming data. We propose online graph filter design which updates the filter online, based on incoming nodes. We design this both for scenarios where the incoming node connectivity is known and unknown. In the unknown connectivity case, we study the performance difference between knowing and not knowing the topology and how the stochastic attachment influences it. We also show that by adapting the stochastic attachment, we can learn faster from the data stream.
Finally,we consider the task of topology decomposition and identification for dynamic networks with fixed nodes but changing edge support. We build a tensor of partially observed adjacency matrices corresponding to such a dynamic topology and express this in terms of underlying latent graphs and their temporal signatures. Furthermore, we account for the time-varying graph signals as a prior to aid identifying these latent graphs and missing components of the topology. These latent graphs are individually and collectively expressive and provide interpretable decompositions along with outperforming traditional structure agnostic low-rank decompositions.

Efficient and effective information retrieval (IR) systems are needed to fetch a large number of relevant documents and present them based on their relevance to the input queries. Previous work reported the use of sparse and dense retrievers. Sparse retrievers offer low latency but suffer from term mismatch issues, while dense retrievers improve performance at the cost of higher processing times. The literature proposed Fast-Forward Indexes, an interpolation-based re-ranking framework that leverages the benefits of both sparse and dense retrievers.

Although a lot of work was done in the field, most studies evaluate the performance of the proposed models only on the MS Marco dataset, neglecting other datasets. This study extends previous work by exploring how different sparse retrievers, employing no-encoder, uni-encoder, and bi-encoder architectures, perform in an interpolation-based re-ranking setting on datasets originating from various domains. Results show that bi-encoder-based retrievers outperform the other sparse retrievers in terms of recall but with a substantial increase in latency compared to simpler retrievers, which generally showed good performance. Further, when the retrievers were used in an interpolation-based re-ranking setting, they performed similarly in terms of ranking quality.

The research explores the impact of rank fusion functions within the retrieve-and-rerank framework with Fast-Forward Indexes. Using the BM25 sparse model for retrieval and TCT-ColBERT dense model for semantic score computation, various rank fusion functions are experimented for the interpolation stage. The interpolated rank in relation to the semantic and lexical ranks is explored. Parametric approach allows to easily adjust the influence of sparse and dense models on the final rank. On the other hand, non-parametric approach lacks flexibility and maintains an equal weight of sparse and dense scores by default. Moreover, the ranking effectiveness and latency are measured to further evaluate each function. Due to flexibility of parametric functions, convex rank fusion function and its normalized variants yield the best trade-off in latency and ranking effectiveness followed by reciprocal rank fusion. On the contrary, non-parametric functions, namely Inverse Square Rank Reciprocal, combMNZ, and Condorcet Fuse, generally performs worse.

In the field of machine learning (ML), the goal is to leverage algorithmic models to generate predictions, transforming raw input data into valuable insights. However, the ML pipeline, consisting of input data, models, and output data, is susceptible to various vulnerabilities and attacks. These attacks include re-identification, attribute inference, membership inference, and model inversion attacks, all posing threats to individual privacy. This thesis specifically targets attribute inference attacks, wherein adversaries seek to infer sensitive information about target individuals.

The literature on privacy-preserving techniques explores various perturbative approaches, including obfuscation, randomization, and differential privacy, to mitigate privacy attacks. While these methods have shown effectiveness, conventional perturbation based techniques often offer generic protection, lacking the nuance needed to preserve specific utility and accuracy. These conventional techniques are typically purpose unaware, meaning they modify data to protect privacy while maintaining general data usefulness. Recently, there has been a growing interest in purpose-aware techniques.
The thesis introduces purpose-aware privacy preservation in the form of a conceptual framework. This approach involves tailoring data modifications to serve specific purposes and implementing changes orthogonal to relevant features. We aim to protect user privacy without compromising utility. We focus on two key applications within the ML spectrum: recommender systems and machine learning classifiers. The objective is to protect these applications against potential privacy attacks, addressing vulnerabilities in both input data and output data (i.e., predictions).

We structure the thesis into two parts, each addressing distinct challenges in the ML pipeline.
Part 1 tackles attacks on input data, exploring methods to protect sensitive information while maintaining the accuracy of ML models, specifically in recommender systems. Firstly, we explore an attack scenario in which an adversary can acquire the user-item matrix and aims to infer privacy-sensitive information. We assume that the adversary has a gender classifier that is pre-trained on unprotected data. The objective of the adversary is to infer the gender of target individuals. We propose personalized blurring (PerBlur), a personalization-based approach to gender obfuscation that aims to protect user privacy while maintaining the recommendation quality. We demonstrate that recommender system algorithms trained on obfuscated data perform comparably to those trained on the original user-item matrix.
Furthermore, our approach not only prevents classifiers from predicting users' gender based on the obfuscated data but also achieves diversity through the recommendation of (non-stereotypical) diverse items. Secondly, we investigate an attack scenario in which an adversary has access to a user-item matrix and aims to exploit the user preference values that it contains. The objective of the adversary is to infer the preferences of individual users. We propose Shuffle-NNN, a data masking-based approach that aims to hide the preferences of users for individual items while maintaining the relative performance of recommendation algorithms. We demonstrate that Shuffle-NNN provides evidence of what information should be retained and what can be removed from the user-item matrix. Shuffle-NNN has great potential for data release, such as in data science challenges.

Part 2 investigates attacks on output data, focusing on model inversion attacks aimed at predictions from machine learning classifiers and examining potential privacy risks associated with recommender system outputs. Firstly, we explore a scenario where an adversary attempts to infer individuals' sensitive information by querying a machine learning model and receiving output predictions. We investigate various attack models and identify a potential risk of sensitive information leakage when the target model is trained on original data. To mitigate this risk, we propose to replace the original training data with protected data using synthetic training data + privacy-preserving techniques. We show that the target model trained on protected data achieves performance comparable to the target model trained on original data. We demonstrate that by using privacy-preserving techniques on synthetic training data, we observe a small reduction in the success of certain model inversion attacks measured over a group of target individuals. Secondly, we explore an attack scenario in which the adversary seeks to infer users' sensitive information by intercepting recommendations provided by a recommender system to a set of users. Our goal is to gain insight into possible unintended consequences of using user attributes as side information in context-aware recommender systems. We study the extent to which personal attributes of a user can be inferred from a list of recommendations to that user. We find that both standard recommenders and context-aware recommenders leak personal user information into the recommendation lists.
We demonstrate that using user attributes in context-aware recommendations yields a small gain in accuracy. However, the benefit of this gain is distributed unevenly among users and it sacrifices coverage and diversity. This leads us to question the actual value of side information and the need to ensure that there are no hidden `side effects'.

The final chapter of the thesis summarizes our findings. It provides recommendations for future research directions which we think are promising for further exploring and promoting the use of purpose-aware privacy-preserving data for ML predictions.

Remote communication applications have become a necessity in a globalised and connectedworld exemplified by the popularity of video conferencing applications. In recent years, virtual reality (VR) remote communication applications have emerged that aim to deliver a greater sense of co-presence and immersion in a shared virtual space where users are able to navigate freely while employing both verbal and non-verbal communication. Such applications require a volumetric user representation and point clouds have emerged as a popular format to represent real-time user reconstructions. However, volumetric point clouds are challenging to deliver over bandwidth-limited networks owing to the large volume of data required for dynamic streams. Adaptive streaming is the process of segmenting an object spatially and temporally in order to optimize the delivery of content by prioritizing the quality of spatial segments that are visible from a given viewport. In this thesis, we embed our research in the application scenario of VR remote communication with real-time point cloud user reconstructions and explore adaptive delivery optimizations. Previous work in the field mainly focused on using the entire point cloud object as the unit of bandwidth allocation in scenes containing multiple point clouds. Other recent work has relied on computationally complex surface estimation in order to spatially segment the point cloud that is unsuited to real-time applications. This thesis focuses on investigating if a user-centred approach combined with a low complexity adaptive streaming method can improve the quality of experience of interacting with point cloud user reconstructions in VR remote communication. We focus on optimizing the delivery of a remote user’s reconstruction with spatial segmentation and surface orientation estimation in real-time combinedwith an auxiliary utility function to allocate the available bandwidth across available segments. The utility is defined based on the position and surface orientation of the point cloud segment and the position and orientation of the user’s viewport.

Self-tracking has expanded exponentially in an era defined by the ubiquitous presence of wearable technologies and smart devices. From health and fitness to finances and productivity, these devices empower users to delve into their quantified self (QS) through an almost infinite amount of visualizations. However, a user has limited time to engage with the data, and the device with which they interact has limited screen space — this calls for the need to suggest the user proactively with valuable and informative plots. An algorithm can suggest and select plots of user activities and adapts to a user’s changing requirements while offering maximum usefulness in the information. We leverage combinatorial optimization to handle the multi-objective task of extracting the most informative 𝑘 plots from a much larger pool of 𝑁 plots. The novel optimization formulation encapsulates plot features into four unique objective functions designed to capture diverse aspects of data usefulness. We evaluate the efficacy of our selection method in silico for various diverse usage scenarios. The simulation results show that the proposed methodology is efficacious in realizing three objectives (‘relevance’, ‘freshness’, and ‘likability’) while identifying the need for refinement in the fourth objective (‘noteworthiness’). Our work demonstrates the design, development, and evaluation of a selection algorithm that delivers a relevant yet fresh selection of visualizations for a quantified-self user interested in keeping track of information related to several activities.

Artists that create mirror anamorphosis usually relies on the principle of trial-and-error and manual mathematical calculations. they have to draw a deformed image on the surface to get the desired reflected image on the mirror object. A drawing interface to make this kind of art does not exist currently. In this paper, we make a suitable drawing interface that satisfies the needs of artists. We analyzed the common needs of artists and propose methods to address them efficiently. Anyone can easily produce mirror anamorphosis with our tool as long they provide the mapping between surface image and reflection image.

Minimalistic, low-poly images have gotten more popular in recent years. But making these low-poly images by hand can be time intensive. For this reason, a lot of research has gone into how to triangulate an image (semi-)automatically. In previous works, only constant color and bilinear interpolation have been used to color the triangles in the triangulation. In this paper, different coloring methods have been investigated and showcased on a fixed mesh. They show some promising first results, but these methods can still be improved upon. When that happens, these coloring methods, combined with previous works on how to optimally triangulate, may result in more accurate or visually pleasing low-poly images.

This paper accelerates the rendering of diffusion curves using the ray tracing cores found on modern NVIDIA graphics cards using the method described by Bowers et al. [Bowers, Leahey and Wang 2011]. This method approximates the final result of the Poisson equation and in this paper is accelerated using Optix and dedicated ray tracing hardware. Using this method yielded a render time decrease of around 8 times between the Quadro P1000 and RTX 2060, while retaining the complex colour gradients and ease of use of traditional diffusion curves solved using the Poisson equation.

This research builds upon a previous method of vectorisation of pixel art by pixel neighbour connecting and boundary energy minimisation. The downside of the previous method is the lack of user input throughout the process and possible divergence between the results and the artist’s vision. The proposed method uses the first part of previous method to connect neighbour pixels and continues with a spring simulation of the boundaries. A process that can be heavily user-guided with adjusting the stiffness of the springs. Results show the proposed method to achieve higher variance in the results, possibly resulting in output images closer to the artist’s vision. However, introduce lengthy process for the user that can be removed by providing higher level GUI.

Many widely used Recommender System algorithms estimate user tastes without accounting for their evolving nature. In recent years there has been a gradual increase in methods incorporating such temporal dynamics through sequential processing of user consumption histories. Some works have also included additional temporal features such as time stamps and intervals between a given user’s interactions with the platform. The latter, in particular, may be a strong signal providing additional context with respect to the current user preferences. However, in previous works this source of information has only been used passingly, without any significant analysis of its impact on recommendation. In this thesis we examine the effects of such intervals, termed time gaps, on recommendation accuracy. In order to do so, we propose a family of novel DeepTimeDelta models, extending a state-of-the-art sequential Recurrent Neural Network based recommender. Through the comparison of our time-dependent models to the sequential baseline we demonstrate that the use of time gaps leads to improvements in recommendation performance, in particular for cases following longer user inactivity. Furthermore, we examine the mechanisms regulating the model recommendation behaviour. Our results suggest that the above performance improvements may be achieved through increased reliance on user long term preferences as well as strong regulation of the importance of the recently consumed items. Finally, we examine the performance differences for users groups with distinct consumption behaviours, demonstrating some improvement for groups featuring less active users as well as users consuming more popular content.

This report describes the implementation of a custom streaming solution from an IP camera to a web browser. The system aims to both provide live video and Video on Demand. This will be used to monitor premature babies in incubators.

Multimedia systems are typically optimized in a way that maximizes users’ satisfaction of using the systems/services. This user satisfaction is what is commonly referred to as Quality of Experience (QoE). For visual media, such as images and videos, the optimization of QoE has meant reducing the visibility of artifacts (e.g. noise or other disturbing factors) in the visual media. This is based on the assumption that the sole appearance of artifacts would disrupt the whole visual experience, in a world where media weremostly consumed passively, and in well defined contexts (e.g., TV broadcasts). Nowadays, the way users experience visual media has changed, thanks to the diffusion of mobile, interactive, immersive, and on-demand technology. Media are now consumed in many different contexts, for example, in the interactive and customizable contexts of social media, or in the immersive contexts of virtual and augmented reality. As consequence of these developments, a user’s visual QoE is no longer determined solely on the appearance of artifacts, but also by factors relevant to the viewing context. This thesis brings in new insights in modeling and automatically assessing users’ visual QoE in view of the developments above. The thesis starts with looking into subjective methodologies for QoE assessments, and continues with developing objective quality metrics that incorporate QoE influencing factors to improve state-of-the-art metrics. Developing reliable and accurate objective metrics to automatically assess users’ visual QoE requires subjective data that are reliable as well. This thesis argues that existing methodologies for collecting subjective data might not be reliable when used to evaluate QoE factors that are highly subjective, or that are new to the research community. Highly subjective quantities may yield different conclusions across experiments. As for new types of media, they often bring the uncertainty on how to evaluate them. Two studies are then presented in this regard. The first study considers the assessment of image aesthetic appeal, as one example of a highly subjective quantity. A large scale study was conducted to compare the use of different subjective methodologies to collect aesthetic appeal data, and some ways to measure the data reliably were proposed. The second study considers the assessment of point cloud quality, as one example of a new type of media (i.e. immersive media). The study explores quantitative and qualitative approaches to understand the way users judge point cloud images. Following the studies on subjective QoE assessments, two studies on objective QoE metrics are presented in this thesis. Despite existing efforts to model the influence of different factors on visual QoE, limited work have proposed to incorporate these factors into existing objective qualitymetrics to improve state-of-the-art. The first study on objective QoE metrics in this thesis investigates the influence of image content/semantic categories (i.e. scene and object categories) on visual QoE, and proposes to include semantic category features in objective image quality metrics. The proposed approach shows improvement from state-of-the-art in predicting image quality. The next study on objective quality metrics investigates new QoE influencing factors for point cloud images, and proposes to incorporate these into an objective quality metric for point cloud images. The results of the studies presented in this thesis show how existing subjective methodologies could yield reliable aesthetic appeal data, and explore point cloud QoE influencing factors. Moreover, the results show that incorporating new QoE influencing factors into objective image quality metrics could improve state-of-the-art performance in predicting users’ QoE. At the end of this thesis, some recommendations are given for future research following up the findings in this thesis.

The twenty-first century has brought plentiful computational power and bandwidth to the masses and has opened up access to multimedia recording devices for everyone. With these developments, a shift in the landscape of multimedia took place: from traditional one-to-many programming (the paradigm of traditional television) to many-to-many creation of diverse content. Nowadays, everyone can become a content creator and connect with new audiences, which has resulted in an explosion of diverse and available multimedia content. In tandem with this change, user needs have evolved as well. Yet, existing multimedia retrieval systems have been struggling to keep up with what users are looking for.

In this thesis, we argue that a multi-perspective approach is desired in order to cater to a diverse range of user needs. In order to know which perspectives should be taken, we turn to the crowd as a source of information on which perspectives would be actually helpful for serving users of multimedia retrieval systems. The central question underlying the research presented in this thesis is: How can we incorporate these perspectives of the crowd into multimedia retrieval systems?

The first major part of the thesis consists of the development of methodologies for effectively addressing the crowd in crowdsourcing studies. It first introduces the concept of framing. Framing allows people to picture a particular scenario that helps them to understand the task at hand and thus would result in high quality answers. Following the framing methodology, the focus shifts to the refinement of elicitation techniques in order to effectively model the common understanding on a particular topic. The methodologies presented in this first part are shown to be useful in informing the design of new features for a multimedia retrieval system.

The second major part of the thesis builds upon the methodologies developed in the first part and uses them to push the research on non-linear video access, i.e., supporting users in consuming relevant parts of a video, further in two ways. First, in a carefully designed crowdsourcing experiment, user comments referring to specifically mentioned time-points in a video are analyzed to build a crowd-informed typology that captures new dimensions of relevance at the time-code level. The usefulness of this typology is tested through a crowdsourced user study on a simulated search scenario. Second, a methodology is developed for obtaining realistic viewing behaviors through crowdsourcing experiments, which can be used in designing and testing new non-linear video access methods. This methodology stresses the importance of not only properly framing the crowdsourcing task, but also that the crowd and multimedia domain are jointly chosen in order to observe behavior that resembles behavior that participants would normally exhibit outside of the experiment. The methodology is used to demonstrate its ability to capture implicit viewing behavior that can be used to support users in non-linearly accessing videos.

The final contributions of the thesis consist of practical pointers for future work and a set of open research questions pertaining crowdsourcing tasks with an interpretive nature. The practical pointers for future work are fueled by experience gained through the various crowdsourcing campaigns that have been carried out throughout the thesis. Addressing these pointers will help in making crowdsourcing research more effective and reduce the effort needed in carrying out experiments. The set of open research questions are formulated by positioning this thesis in relation to prior related work. These questions serve as a starting point for future research on interpretive crowdsourcing tasks and pursuing them could aid the development of retrieval systems with multiple perspectives on multimedia.