With the following paper we are planning to present and explore the possibilities of the the newly introduced Poisson Flow Generative Model (PFGM). More specifically, this work aims to introduce the Conditional Poisson Flow Generative Model (CoPFGM), which by extending the existing repository of the PFGM, it will be able to be trained in a way that allows for conditional image sampling. The work aims to provide a more modular solution that can be easily adjusted for multiple data sets, including custom, as well as datasets taken directly from large Python libraries such as PyTorch and TensorFlow. Our proposed CoPFGM consists of two steps: (i) modifying the input of underlying UNet and (ii) modifying the loss function. More specifically, for (i) we have augmented the input channels of every image with one-hot-like class conditional images, and about (ii) we are introducing an updated loss function which incorporates the Cross-Entropy Loss of the generated images during training. The proposed model is tested against 2 datasets, the MNIST, and the Dilbert dataset, the latter, consists of 1100 custom images of the faces of 6 characters taken from the Dilbert Comic-Strip. The proposed model will be tested and presented in the form of an Ablation Study, with which, we show the conditional behavior of the channel augmentation, and the image improvement in terms of class representation with the Cross-Entropy loss.

Hyperparameter optimization(HPO) forms a critical aspect for machine learning applications to attain superior performance. BOHB (Bayesian Optimization and HyperBand) is a state of the art HPO algorithm that approaches HPO in a multi-armed bandit strategy, augmented with Bayesian optimization to drive configuration sampling. However, BOHB requires predefined distribution of fidelities for each tuning task. The challenge in this is that it is impossible to define fidelities a priori, since each machine learning model is uniquely complex and requires different amount of compute resources for convergence. Furthermore, in our empirical analysis, we found that each HPO task rendered different performance trajectories on different fidelity (budget) types. Thus, the challenge of defining fidelities also extends to choosing an optimal budget type. To alleviate these challenges, we present MultiTune: a budget allocation scheme that builds on top of BOHB to dynamically define fidelities for optimization. MultiTune incorporates an algorithm to dynamically choose a preferred budget type for an HPO task, coupled with 2D gradient based budget constraint explorations to enable granular definition of fidelities. Through our empirical analysis, we show that MultiTune can consistently converge to a well performing configuration without significant computation overhead.

Deep neural networks (DNNs) are becoming the core components of many applications running on edge devices,especially for image-based analysis, e.g., identifying objects, faces, and genders. While very successful in resource rich environments like the cloud of powerful computers, utilizing Deep Learning on edge devices for inference is not used to great extend. The resource constraints of edge devices present bottlenecks that prevent all but small networks from on-device inference. Existing solutions like compression or off-loading to the cloud sacrifice either model accuracy or inference latency. To counter the aforementioned shortcomings, Masa and EdgeCaffe are proposed as a solution. The process of DNN inference by existing frameworks is to loaded and execute a model in its entirety. As this ignores the resource limitations that are inherent to edge devices, a new framework EdgeCaffe is proposed. EdgeCaffe is able to execute models partially and thereby reducing the memory footprint of the model. Additional, EdgeCaffe allows for coordination between models when a multi-model DNN inference is executed. Masa , a memory-aware multi-DNN scheduling algorithm, featuring on modeling inter-and intra-network dependency and leveraging complimentary memory usage of each layer. Masa can consistently ensure the average response time when deterministically and stochastically executing multiple DNN-based image analysis. With extensive evaluations, Masa can consistently ensure the average response time when deterministically and stochastically executing multiple DNN-based image analyses. We extensively evaluate Masa on three configurations of Raspberry Pi and a large set of popular DNN models triggered by different arrival patterns of images. Our evaluation results show that Masa can achieve lower average response times by up to 8× and 16× for deterministic and stochastic arrivals respectively on devices with small memory, i.e., 512 MB to 1 GB, compared to the state of the art multi-DNN scheduling solutions.

Synthesizing audio-reactive videos to accompany music is challenging multi-domain task that requires both a visual synthesis skill-set and an understanding of musical information extraction. In recent years a new flexible class of visual synthesis methods has gained popularity: generative adversarial networks. These deep neural networks can be trained to reproduce arbitrary images based on a dataset of about 10000 examples. After training, they can be harnessed to synthesize audio-reactive videos by constructing sequences of inputs based on musical information. Current approaches suffer from a few problems which hamper the quality and usability of GAN-based audio-reactive video synthesis. Some approaches consider only a small number of possible musical inputs and ways of mapping these the GAN's parameters. This leads to weak audio-reactivity which has a similar motion characteristic across all musical inputs. Other approaches do harness the full design space, but are difficult to configure correctly for effective results. This thesis aims to address the tradeoff between audio-reactive flexibility and ease of attaining effective results. We introduce multiple algorithms that explore the design space by using machine learning to generate sequences of inputs for the GAN. To develop these machine learning algorithms, we first introduce a metric, the audiovisual correlation, that measures the audio-reactivity in a video. We use this metric to train models based only on a dataset of audio examples, avoiding the need of a large dataset of example audio-reactive videos. This self-supervised approach can even be extended to optimize a single audio-reactive video directly, removing the need to even train a model beforehand. Our evaluation of the methods shows that our algorithms out-perform prior work in terms of their audio-reactivity. Our solutions explore a wider range of the audio-reactive space and do so without the need for manual feature extraction or configuration.

In the use of Machine Learning systems, attaining the trust of those that are the end-users can often be difficult. Many of the current state-of-the-art systems operate as Black-Boxes. Errors produced by these Black-Box systems, without further explanation as to why these decisions were made, will deteriorate trust. This effect is especially strong when these erroneous decisions are generated with high confidence. This thesis presents both a data-driven as well as a human-in-the-Loop based methodology to characterize and mitigate high-confidence errors. We propose an Iterative Expert Session based methodology. By engaging domain experts through a series of interaction sessions, we aim to reduce the disconnect and knowledge gap between data scientists and domain experts, and to ultimately increase trust in the model. A practical approach was taken working in close connection with the practice of the data scientists of the ILT, helping them in improving their model and providing a direct contribution. We study the problem in the context of Road and Transportation law violations, by engaging inspectors (i.e., domain experts) in day-in-the-life and in-house interview sessions. A thorough analysis is performed of the most important features for data instances that were in error with a high degree of confidence. A method is presented that helps in characterizing these errors by predicting errors. We show that by careful removal of biased data features, proper data selection and by bridging the knowledge gap between domain experts and data scientists, we can improve the performance of the machine learning model. We show an increase of model Precision from 0.56800 with a baseline of 0.32968 to a Precision of 0.52077 with a baseline of 0.23473. Considering the baseline, this is an increase of 28.9% in Precision. We reduce biases existent in the data by reducing variables that predict on inspector practice. The magnitude of High Confidence Errors in the top 20% errors went from 0.70435 to 0.70465 showing an improvement taking into account the reduced baseline and removal of overfitted variables.

Federated learning enables the construction of machine learning models, while adhering to privacy constraints and without sharing data between different devices. It is achieved by creating a machine learning model on each device that contains data, and then combining these models through an aggregation algorithm without sharing the data. Federated learning is currently a hot topic, and a lot of research has gone into implementing accurate aggregation algorithms. The original algorithm is FedAvg, and since then many different algorithms have been introduced. In this paper, I will compare the performance of five different aggregation algorithms: FedAvg, FedProx, FedYogi, FedMedian and q-FedAvg. The algorithms are compared on different data sets, namely MNIST and a kinase inhibition data set, as well as on different data distributions and number of clients. The experiments indicate that among these five algorithms, FedYogi achieves the best performance, both in terms of highest final accuracy as well as in terms of convergence rate.

Synthetic art is the end result of artificial intelligence models that have been trained to generate images from text prompts. "Comic synthesis" is one such use case, where comic illustrations are produced from textual descriptions. Previous attempts at comic synthesis have utilized conditional Generative Adversarial Networks (cGANs), but this approach has encountered challenges in generating consistent and visually appealing comic panels. Strict data requirements and quality limitations have left room for improvement. We propose a novel approach to comic synthesis using Stable Diffusion, a powerful generative modelling technique. The study investigates the fine-tuning of the stable diffusion model specifically for the generation of Dilbert Comics from textual prompts. We explore different techniques to fine-tune the stable diffusion model for comic synthesis including Dreambooth and LoRA. Through extensive experimentation and analysis, with an FID score of 123, results produced using the Lora technique outperformed Dreambooth, excelling in understanding the Dilbert style, while Dreambooth struggled with multiple-subject training. Results are also compared with previous approaches based on conditional GANs. While the quality and detail greatly improved, the transition from conditionals to text descriptions meant the results were less accurate. The results show the potential of stable diffusion in generating appealing Dilbert Comic panels while highlighting the need for further exploration to enhance the alignment between textual descriptions and the generated images.

Federated Learning is a machine learning paradigm for decentralized training over different clients. The training happens in rounds where each client learns a specific model which is then aggregated by a central server and passed back to the clients. Since the paradigm’s inception, many frameworks that provide Federated Learning tools and infrastructure have appeared. This leads to the question of ”How do different Federated Learning frameworks compare?”, which is the research question of this paper. The paper’s main contribution will be helping developers new to the Federated Learning field decide between NVidia Flare, OpenFL, and Flower, three popular federated learning frameworks.

Face clustering is a subfield of computer vision and pattern recognition with many applications such as face recognition and surveillance. Accurate clustering of faces can also help us to create labeled datasets. However, in the domain of comics, face clustering is not well studied. Therefore, it is uncertain which methods of feature extraction and clustering perform well on faces of comic characters. In this paper, we investigate the effectiveness of comic face clustering. To conduct our investigation, we implement two pipelines: one that automatically extracts character faces from comic strips, and another that clusters the extracted faces. Using Dilbert Comics for our experiments, we examine the performance of various feature extraction and clustering methods. Additionally, we experiment with combining feature extraction methods and removing noisy samples to increase the clustering accuracy. We show that using color information is crucial for accurate clustering, and combining color with shape features further improves accuracy. However, our experiments indicate that accuracy improvement is not guaranteed for every combination of feature extraction methods. We also demonstrate that removing noisy samples using hierarchical clustering can increase clustering precision. Using our findings, we achieve an F1 score of 0.752 based on our Dilbert Comics dataset of 77,768 face images. We obtain this result by clustering 20,988 non-noisy face images into 35 clusters with a precision of 0.886.

Sharing data is becoming increasingly difficult, due to the regulatory constraints imposed by the General Data Protection Regulation (GDPR). Businesses are not allowed to share data which contains privacy sensitive information. Synthetic data generation has emerged as a solution to this problem. State of the art generative adversarial networks (GAN) can generate synthetic data which statistically resembles the original data, while changing privacy sensitive information so that it cannot be related back to a person. However, the process of generating synthetic data is still a very time consuming process for data scientists. One of the challenges faced in synthetic data generation is aptly modeling the raw data; transforming it into numerical, and specifying the hyper-parameters such as which columns are categorical, mixed type, numerical or log distributed, is a non-trivial task. Another challenge is making estimations about the underlying distributions of the data and how these different distributions are correlated. The proposed solution UniformGAN addresses these issues by adopting a transformer which can handle raw data and detect the data type and transforms it into a numerical equivalent. It uses the data type and estimated distribution to set the hyper-parameters for categorical columns, mixed columns, and log columns. Furthermore, it estimates the underlying distributions of the data and leverages a statistical transformation in order for the machine learning model to easier learn the dependence structure of variables. The evaluation with regard to machine learning utility, statistical similarity, and privacy preverabiliy has shown that UniformGAN improves accuracy with regard to decision tree classification utility, improving averaged machine learning utility by 2% compared to CTAB-GAN, and 19.21% compared to copulaGAN, while maintaining statistical similarity and privacy preservability compared to state of the art tabular data modeling techniques.

Federated learning (FL) has emerged as a promis-ing approach for training machine learning models using geographically distributed data. This paper presents a comprehensive comparative study of var-ious machine learning models in the context of FL. The aim is to evaluate the efficacy of these models in different data distribution scenarios and provide practical insights for practitioners in the field. The findings highlight the performance and limitations of linear and non-linear models on MNIST and Ki-nase datasets.

Few-shot learning presents the challenging problem of learning a task with only a few provided examples. Gradient-Based Meta-Learners (GBML) offer a solution for learning such few-shot problems. These learners approach the few-shot problem by learning an initial parameterization that requires only a few adaptation steps for new tasks. Although these GMBLs are well-studied with correct training data, few have studied the impact of training them with noisy labels. In this thesis, we show that GMBLs are negatively affected by label noise. We propose a training strategy (BatMan-CLR) leveraging a novel subsampling approach to address the impact of meta-training with label noise. To train and evaluate the different GMBLs, we implement nmfw, a novel framework for extensible training loop definition and few-shot data generation. Our results show that BatMan-CLR is capable of learning few-shot classification models. We show that our approach can effectively mitigate the impact of meta-training label noise. Even with 60% wrong labels BatMan and Man can limit the meta-testing accuracy drop to 2.5, 9.4, and 1.1 percent points, respectively, with existing meta-learners across the Omniglot, CifarFS, and MiniImagenet datasets.

Federated learning allows multiple parties to collaboratively develop a deep learning model, without sharing private data. Models can be generated from the most up-to-date data while taking unique and not publicly available data into account. However, the distributed nature of federated learning causes problems too, and clients are not guaranteed to hold independently identically distributed (iid) data, causing performance degradation. This work analyzes existing methods of generating such skewed datasets and finds that the Earth Movers Distance (EMD) can be used to compare them. A novel scheme called phase-shift is introduced, which allows clients to communicate more frequently, without increasing communication, hereby reducing drift caused by non-iid data. Finally, we propose a data-driven approach that can reduce the data skew by supplementing local datasets with augmented data. A novel method of balancing unaltered and augmented data is introduced, taking the skew of the dataset into account. Empirical analysis shows that phase-shift can reduce the instantaneous communication load on the system by 37.5% without suffering a performance loss or reducing convergence rate. Evaluation of data augmentation on a heavily skewed cifar10 dataset shows that accuracy is improved by 10%. Finally, phase-shift and data augmentation are combined, resulting in a 13% accuracy improvement, surpassing algorithms such as FedNova and FedProx when dealing with label-heterogeneity.

Serverless computing is a relatively recent paradigm that promises fine-grained billing and ease-of-use by abstracting away cloud infrastructure for developers. There is an increasing interest in using the serverless paradigm to execute data analysis tasks. Serverless functions often interact with external services, which can be considered similar to the concept of side-effects in regular programming. Haskell uses monads to isolate side-effects and to structure the composition of functions using side-effects. This thesis explores whether the concept of the monad can be applied in a serverless computing environment, ideally in a way that is flexible with regards to platform. An abstraction of side-effects was developed in the form of a monadic layer that is added to serverless functions. The monadic layer interacts with monads using an interface and exposes the API of the monads to the user. A monadic implementation was also created for a platform-independent function composition mechanism using orchestrator functions. An implementation of a shared state side-effect has also been created as a practical use-case for the monadic layer, and to explore the usability of monads on platforms with more restricted composition frameworks. The implementations are evaluated on performance, expressiveness and usability.

Payment channels allow parties to utilize the blockchain to send transactions for a cheaper fee. Previous work has analyzed to which degree a party can profit by facilitating the transaction process. The aim is to increase the usability of the network and to be rewarded for providing this service. However, previous work focuses on maximizing the reward of the individual player in isolation, a model that we aim to expand. That is why in this work we extend the action space to allow other parties to act and react, and observe the impact this has on the rewards of the player that would otherwise act in isolation. Testing existing placement strategies by performing channel placement games, we can assess the difference in the reward that indicates the potential loss that competition may cause when operating in the Bitcoin Lightning Network. Furthermore, we have developed a new strategy that is able to improve the performance in the multi-actor model.

Contrastive Language-Image Pretraining (CLIP) has gained vast interest due to its impressive performance on a variety of computer vision tasks: image classification, image retrieval, action recognition, feature extraction, and more. The model learns to associate images with their descriptions, a powerful method which allows it to perform well on unseen domains. Often, the descriptions fail to capture text which is contained within the image, a source of information which could prove useful for a handful of computer vision tasks. This limitation requires finetuning in domains where contained text is important. In fact, CLIP has mixed performance on Optical Character Recognition (OCR). This paper proposes a novel architecture: OSBC (OCR Sentence BERT CLIP), which combines CLIP and a custom text extraction pipeline, composed of an OCR model, and a Natural Language Processing (NLP) model. OSBC uses the text contained within images as an additional feature when performing image classification and retrieval. We tested the model on multiple datasets for each task, occasionally outperforming CLIP when images contained text, while maintaining finetunability, and improving the model's robustness. In addition, OSBC was designed to be generalizable, meaning it is expected to perform well on unseen domains without finetuning, though this was not achieved in practice.

Multi-Server Federated Learning (MSFL) is a decentralised way to train a global model, taking a significant step toward enhanced privacy preservation while minimizing communication costs through the use of edge servers with overlapping reaches. In this context, the FedMes algorithm facilitates the aggregation of gradients, contributing to the convergence of the global model. Attacks that aim to reduce the accuracy of the global model are called untargeted attacks. One such attack that is particularly difficult to detect is the Min-Max attack. This paper explores the extension of existing defenses to enhance the robustness of MSFL against the Min-Max attack. To do this, existing state-of-the-art defenses, including Median, Krum, Multi-Krum, Trimmed-Mean, Bulyan and DnC are extended and examined for their adaptability to this context. We refer to the extended versions of these defenses as FMes-Defenses. Our results indicate that FMes-Defenses are ineffective in preventing the Min-Max attack from diminishing the accuracy of the global model. Surprisingly, we find even FMes-DnC is inadequate despite it's Single-Server counterpart (DnC) being renowned for mitigating the Min-Max attack. These findings emphasise the need for novel defenses specifically tailored to the nuances of MSFL. While representing a significant stride in communication efficiency, MSFL, complemented by the FedMes algorithm, may require additional measures to ensure robust security against sophisticated untargeted attacks. This research contributes valuable insights into the challenges and importance of enhancing the security of MSFL in its ongoing development.

Abstract—Multi-label classification is an important branch of classification problems as in many real world classification scenarios an object can belong to multiple classes simultaneously. Deep learning based classifiers perform well at image classifica- tion but their predictions have shown to be unstable when subject to small input distortions, called adversarial perturbations. There are multi-class classifiers, which assign images to a single class, and multi-label classifiers that attribute multiple labels to an image. In multi-class scenarios these adversarial attacks are conventionally constrained by a perturbation magnitude budget in order to enforce visual imperceptibility. In the related studies concerning multi-label attacks there has been no notion of a budget and this results in visible perturbations in the image. In this paper we develop attacks that cause the most severe disruptions in the binary label predictions, i.e. a maximum number of label flips, while adhering to a perturbation budget. To achieve this, we first analyse the applicability of the existing single label attack MI-FGSM on multi label problems. A naive way of using MI-FGSM in a multi-label scenario means using binary cross entropy loss and targeting all labels simultaneously. Our key observations are that targeting all labels simultaneously when restricted to a small budgets leads to inefficient budget use, that all labels have different attackability and also that labels exhibit different correlation structures which influences the combined attackability. Moreover, we show that the loss function determines the optimisation direction through prioritising labels with certain confidence values. We find that there are two dif- ferent strategies to optimise budget use and propose two distinct methods namely, Smart Loss-function for Attacks on Multi-label models (SLAM) and Classification Landscape Attentive Subset Selection (CLASS). SLAM comprises a loss function that uses an estimate for the potential amount of flips to adapt the shape of the curve, and hence the label prioritisation. CLASS uses binary cross entropy loss but focuses the budget on merely a subset of the labels, which was constructed while considering label attackability and pairwise label correlation. CLASS does have the drawback that it relies on classifier specific heuristics for determining the size of the label subset. We extensively evaluate SLAM and CLASS on three datasets, using two state of the art models, namely Query2Label and ASL. Our evaluation results show that CLASS and SLAM are able to increase the flips given the budget constraint by up to 131% and 61% respectively compared to naive MI-FGSM.