Face clustering is a subfield of computer vision and pattern recognition with many applications such as face recognition and surveillance. Accurate clustering of faces can also help us to create labeled datasets. However, in the domain of comics, face clustering is not well studied. Therefore, it is uncertain which methods of feature extraction and clustering perform well on faces of comic characters. In this paper, we investigate the effectiveness of comic face clustering. To conduct our investigation, we implement two pipelines: one that automatically extracts character faces from comic strips, and another that clusters the extracted faces. Using Dilbert Comics for our experiments, we examine the performance of various feature extraction and clustering methods. Additionally, we experiment with combining feature extraction methods and removing noisy samples to increase the clustering accuracy. We show that using color information is crucial for accurate clustering, and combining color with shape features further improves accuracy. However, our experiments indicate that accuracy improvement is not guaranteed for every combination of feature extraction methods. We also demonstrate that removing noisy samples using hierarchical clustering can increase clustering precision. Using our findings, we achieve an F1 score of 0.752 based on our Dilbert Comics dataset of 77,768 face images. We obtain this result by clustering 20,988 non-noisy face images into 35 clusters with a precision of 0.886. ... Read more

Synthetic art is the end result of artificial intelligence models that have been trained to generate images from text prompts. "Comic synthesis" is one such use case, where comic illustrations are produced from textual descriptions. Previous attempts at comic synthesis have utilized conditional Generative Adversarial Networks (cGANs), but this approach has encountered challenges in generating consistent and visually appealing comic panels. Strict data requirements and quality limitations have left room for improvement. We propose a novel approach to comic synthesis using Stable Diffusion, a powerful generative modelling technique. The study investigates the fine-tuning of the stable diffusion model specifically for the generation of Dilbert Comics from textual prompts. We explore different techniques to fine-tune the stable diffusion model for comic synthesis including Dreambooth and LoRA. Through extensive experimentation and analysis, with an FID score of 123, results produced using the Lora technique outperformed Dreambooth, excelling in understanding the Dilbert style, while Dreambooth struggled with multiple-subject training. Results are also compared with previous approaches based on conditional GANs. While the quality and detail greatly improved, the transition from conditionals to text descriptions meant the results were less accurate. The results show the potential of stable diffusion in generating appealing Dilbert Comic panels while highlighting the need for further exploration to enhance the alignment between textual descriptions and the generated images. ... Read more

In the past decade data-driven approaches have been at the core of many business and research models. In critical domains such as healthcare and banking, data privacy issues are very stringent. Synthetic tabular data is an emerging solution to privacy guarantee concerns. Generative Adversarial Networks (GANs) are one of the emerging solutions for synthesizing data. However in order to capture all relevant relationships between columns, tabular data needs to be numerically encoded. As columns might be of different types, this is a challenging task as proven by recent approaches. Throughout this paper, we focus on the dimensionality explosion problem, which leads to high-dimensional datasets alongside computational overhead and increase in training time. We introduce a novel synthesis pipeline - LCT-GAN - an improvement to the current state-of-the-art in tabular data synthesis CTAB-GAN. Our approach addresses the dimensionality explosion problem by introducing a low-dimensional embedding step via an autoencoder prior to training. It is then combined with a novel conditional GAN architecture, operating in latent space. After thorough evaluation, we observe that our solution achieves more than 30\% improvement in certain statistical metrics in comparsion to CTAB-GAN, accompanied by 5 fold decrease in size and 150 times speedup in training time for a single epoch. We successfully show that it is possible to embed data using autoencoders, and that GANs are able to learn complex relationships in latent space in the context of tabular data. ... Read more

Since the regularization of data privacy (e.g., GDPR), the effectiveness of data sharing has decreased. A promising technique to circumvent this problem is tabular data synthesis (i.e., the generation of fake tabular data that statistically resembles the original data). However, the state-of-the-art tabular data synthesis model, CTAB-GAN, fails at robustly imitating global data dependencies and underperforms when column orders get permuted. CTAB-GAN internally uses Convolutional Neural Networks (CNN) which limits the model’s performance due to a strictly non-global data perspective during iterative training phases. To address this limitation, this paper proposes FCT-GAN which leverages the Fourier Neural Operator to learn global dependencies in the frequency domain. Specifically, it enhances CTAB-GAN by replacing the CNN of the discriminator with a four-layered two-dimensional Fourier Neural Operator. As a consequence of FCT-GAN’s global nature and cross-column relation robustness, it outperforms CTAB-GAN and additionally offers the column permutation invariant property. The evaluation of FCT-GAN on five datasets shows that the generated data, remarkably resembles the real data and reveals an increase in accuracy, by up to 19% for five machine learning algorithms independent of data column order, compared to CTAB-GAN. ... Read more

Sharing data is becoming increasingly difficult, due to the regulatory constraints imposed by the General Data Protection Regulation (GDPR). Businesses are not allowed to share data which contains privacy sensitive information. Synthetic data generation has emerged as a solution to this problem. State of the art generative adversarial networks (GAN) can generate synthetic data which statistically resembles the original data, while changing privacy sensitive information so that it cannot be related back to a person.


However, the process of generating synthetic data is still a very time consuming process for data scientists.


One of the challenges faced in synthetic data generation is aptly modeling the raw data; transforming it into numerical, and specifying the hyper-parameters such as which columns are categorical, mixed type, numerical or log distributed, is a non-trivial task. Another challenge is making estimations about the underlying distributions of the data and how these different distributions are correlated.

The proposed solution UniformGAN addresses these issues by adopting a transformer which can handle raw data and detect the data type and transforms it into a numerical equivalent. It uses the data type and estimated distribution to set the hyper-parameters for categorical columns, mixed columns, and log columns.
Furthermore, it estimates the underlying distributions of the data and leverages a statistical transformation in order for the machine learning model to easier learn the dependence structure of variables.

The evaluation with regard to machine learning utility, statistical similarity, and privacy preverabiliy has shown that UniformGAN improves accuracy with regard to decision tree classification utility, improving averaged machine learning utility by 2% compared to CTAB-GAN, and 19.21% compared to copulaGAN, while maintaining statistical similarity and privacy preservability compared to state of the art tabular data modeling techniques. ... Read more

While data sharing is crucial for knowledge development, privacy concerns and strict regulation (e.g., European General Data Protection Regulation (GDPR)) unfortunately limit its full effectiveness. Synthetic tabular data emerges as an alternative to enable data sharing while fulfilling regulatory and privacy constraints. The state-of-the-art tabular data synthesizers draw methodologies from Generative Adversarial Networks (GAN). In this thesis, we develop CTAB-GAN, a novel conditional table GAN architecture that can effectively model diverse data types with complex distributions. CTAB-GAN is extensively evaluated with the state of the art GANs that generate synthetic tables, in terms of data similarity and analysis utility. The results on five datasets show that the synthetic data of CTAB-GAN remarkably resembles the real data for all three types of variables and results into higher accuracy for five machine learning algorithms, by up to 17%.

Additionally, to ensure greater security for training tabular GANs against malicious privacy attacks, differential privacy (DP) is studied and used to train CTAB-GAN with strict privacy guarantees. DP-CTAB-GAN is rigorously evaluated using state-of-the-art DP-tabular GANs in terms of data utility and privacy robustness against membership and attribute inference attacks. Our results on three datasets indicate that strict theoretical differential privacy guarantees come only after severely affecting data utility. However, it is shown empirically that these guarantees help provide a stronger defence against privacy attacks. Overall, it is found that DP-CTABGAN is capable of being robust to privacy attacks while maintaining the highest data utility as compared to prior work, by up to 18% in terms of the average precision score. ... Read more

Generative Adversarial Networks are widely used as a tool to generate synthetic data and have previously been applied directly to time-series data. However, relying solely on the binary adversarial loss is not sufficient to ensure the model learns the temporal dynamics of the data. TimeGAN [14] introduces an additional reconstruction and supervised loss to tackle this issue. We have been able to reproduce results similar to those of the original TimeGAN paper [14], after fixing several issues in the provided implementation by the authors of TimeGAN. Furthermore, we propose two novel improvements to the existing algorithm. Firstly we updated the implementation to Tensorflow 2 to ensure compatibility across systems. Secondly by scaling the epochs over the three training phases we are able to reduce the overall training time up to 29\% and produce results equal or better compared to the benchmark. ... Read more

The key to producing high-fidelity time-series data is to preserve temporal dynamics. This means that generated sequences respect the relationship between variables across time as in the original data. While new types of GANs have been used to generate time-series data, they, like previous GAN
implementations, are time consuming to train. A novel federated framework is proposed, which generates realistic time-series data, by combining supervised and unsupervised training. The framework is based on the work in TimeGAN and Federated GAN (FeGAN). Using an embedded learning space, TimeGAN
encourages the network to mimic the structure of the training data. FeGAN allows the results of TimeGAN to be combined at a central server, which has benefits for both throughput, and potential to improve data privacy. This also introduces the possibility of using cross domain data. The challenge with creating applying federated learning to TimeGAN, and timeseries data in general is whether the learned temporal dynamics can be combined. This is accomplished by the combination of the weighting and sampling scheme used. This paper demonstrates, by qualitative and quantitative analysis, the ability novel framework proposed, to produce equivalent quality synthetic timeseries data compared to the original TimeGAN, without sharing local data between nodes in the network. This is based on the predictive and discriminative scores described, as well as PCA and t-SNE analysis. Additionally, there is an approximate eleven percent increase in Floating Point Operations per second when using one machine, and up to a thirty percent increase when using multiple. ... Read more

With a growing need for data comes a growing need for synthetic data. In this work we reproduce the results of DoppelGANger [16] in synthesising time series data with metadata. We identify a key issue in the comparison made in [16] of DoppelGANger to TimeGAN, RNNs, AR and HMM models, which creates a new avenue of time series synthesis using GANs. We show that not all results of [16] can be reproduced. We furthermore find that DoppelGANger does not adequately capture measurement-metadata correlations of our dataset. Sample size reduction is shown to be an effective tool to reduce training time while still attaining accurate results, and the key parameter S is tuned further. Finally we show that execution on CPU has similar training times as execution on GPU by [16], suggesting that the original code can be improved, and we release our version of the models ourselves, to enable easy reproduction. In closing points we shine light on possible future improvements that we were unable to test ourselves, and conclude that DoppelGANger is a promising model that opens the door to new unseen applications of GANs for time series synthesis. ... Read more

Generative Adversarial Networks (GANs) are a modern solution aiming to encourage public sharing of data, even if the data contains inherently private information, by generating synthetic data that looks like, but is not equal to, the data the GAN was trained on. However, GANs are prone to remembering samples from the training data, therefore additional care is needed to guarantee privacy. Differentially Private (DP) GANs offer a solution to this problem by protecting user privacy through a mathematical guarantee, achieved by adding carefully constructed noise at specific points in the training process. A state-of-the-art example of such a GAN is Gradient Sanitized Wasserstein GAN, (GS-WGAN), \cite{chen2021gswgan}. This model is shown to create higher quality synthetic images than other DP GANs. To extend the applicability of GS-WGAN we first reproduce and extend the evaluation, verifying that the model outperforms DP-CGAN by an average of 40\% when assessed across three qualitative metrics and two datasets. Secondly we propose improvements to the architecture and training procedure to make GS-WGAN applicable on timeseries data. The experimental results show that GS-WGAN is fit for generating synthetic timeseries through promising experimental results.

[1] D. Chen, T. Orekondy, and M. Fritz, “Gs-wgan: A gradient-sanitized approach for learning differentially private generators,” 2021 ... Read more

The field of Natural Language Processing (NLP) techniques has progressed rapidly over the recent years. With new advancements in transfer learning and the creation of open-source projects like BERT, solutions and research projects emerged implementing new ideas in a variety of domains, for tasks including text classification or question answering. This research focuses on the task of humor detection and sentiment analysis of comic texts with the use of a fine-tuned BERT model. Never before has a fine-tuned BERT model been used for the task of humor detection in a text coming from an artwork. Comic text features domain-specific language, affecting the meaning and structure of commonly used grammar and vocabulary. This may differ from the language people use every day, and from the language the existing humor classifiers used for training. This research contributes to the NLP field with new models and datasets for humor detection and sentiment analysis, and reports on techniques to improve the training times and the accuracy of a pre-trained BERT model on small datasets. The proposed solution trained on comic datasets outperforms the chosen baselines and could be used as a reliable classifier in the given domain. Moreover, the results indicate that techniques reducing the training time of the model can positively contribute to its performance. ... Read more

Object detection and recognition is a computer vision problem tackled with techniques such as convolutional neural networks or cascade classifiers. This paper tackles the challenge of using the similar methods in the realm of comics strips characters. We approached the idea of combining cascade classifiers with various convolutional neural network architectures for character detection and recognition in consideration of maintaining low computational overhead. The alternative with the selective search algorithm step was also explored. The name of the pipeline is HaarCNN. We compared it to standard methods to verify a potential improvement. We evaluate 750 number of images extracted from comic strips and achieve over 85% precision and around 80% recall of detected faces and over 80% of correct main character recognitions. The images were processed in around 200 seconds. The potentially satisfying results of character annotation can be advantageous in deep learning sub-fields such as generative adversarial networks. ... Read more

Comic illustrations and transcriptions form an attractive dataset for several problems, including computer vision tasks, such as recognizing character’s faces, generating new comics, or natural language processing tasks like automated comic translation or detecting emotion in the dialogues. However, despite a large number of comic strips published online, very few datasets of annotated comic illustrations are available. This forms a bottleneck for further advancements in the field. The source of the data scarcity is the manual labor required for annotation — one has to download the comic strips, separate each strip into panels (individual illustrations), and transcribe the text. Automating the process is needed, but it poses several challenges. Panel detection in comic strips is non-trivial, due to varying layouts and styles of comics. Automated transcription is also challenging, as the out-of-the-box optical character recognition (OCR) models struggle with diverse fonts, hand-writing styles, and backgrounds. We design an automatic comic text-image dataset construction pipeline, termed DCP, consisting of three components: (i) web scraping, (ii) panel extraction, and (iii) text extraction. A multi-threaded comic scraper is created to download all the comics. A panel extraction algorithm based on panel frame detection is developed to divide the comic strips into individual illustrations. Lastly, to effectively extract the text using OCR, we propose additional pre-processing and post-processing steps, namely, up-scaling and binarizing images, clustering-based text ordering, and dictionary-based autocorrect. We extensively evaluate the prototype of DCP on three comic series: PHD Comics, Dilbert, and Garfield. Web scraping is used to downloading over 25000 comic strips at an average pace of 149ms per image. Panel extraction results on 1118 panels show success rates of 100%, 97%, and 71% for Dilbert, PHD Comics and Garfield respectively, outperforming the baseline in terms of accuracy and speed. The text extraction algorithm, tested on 1100 comics, achieves a 7x error reduction compared to the out-of-the-box OCR. ... Read more

The creation of comic illustrations is a complex artistic process resulting in a wide variety of styles, each unique to the artist. Conditional image synthesis refers to the generation of de novo images based on certain preconditions. Applying machine learning to conditionally generate novel comics proves an intriguing yet difficult task. This paper aims to answer whether Generative Adversarial Networks (GANs) can be used for conditional comic synthesis. Recent advancements in Generative Adversarial Networks have increased the capability of image synthesis to hyper-realistic levels. Despite this, the performance of GAN models is almost always assessed on photo-realistic images. To extend experimental knowledge of unconditional GAN performance into the domain of comics, an empirical analysis was performed on the unconditioned generative performance of three cutting edge GAN architectures: Deep Convolutional GAN (DCGAN), Wasserstein GAN (WGAN), and Stability GAN (SGAN). This paper showed that the SGAN implementation far outperforms both the DCGAN and WGAN architectures on a dataset of Dilbert comics, achieving an FID score of 89.1. Due to their relative simplicity, comics provide an intriguing candidate for conditional generation. A comic panel can likely be described using a few specific labels (eg. background and characters). Two conditional networks were created, using the SGAN architecture as a baseline. Multi Class SGAN (MC-SGAN) used a traditional multi-class conditional approach while the Multi Label SGAN (ML-SGAN) utilized a multi-label auxiliary classification approach. Multiple experiments were performed between these two networks resulting in hundreds of hours of training. While performance between the networks was quite similar on simple conditional tasks, on more complex tasks MC-SGAN outperformed ML-SGAN. MC-SGAN was able to conditionally generate comics based on character and color, with desired conditions distinguishable in almost all outputs. Issues with traditional methods of auxiliary classifier training in the MC-SGAN implementation are additionally identified and discussed. ... Read more

Drawing and annotating comic illustrations is a complex and difficult process. No existing machine learning algorithms have been developed to create comic illustrations based on descriptions of illustrations, or the dialogue in comics. Moreover, it is not known if a generative adversarial network (GAN) can generate original comics that correspond to the dialogue and/or descriptions. GANs are successful in producing photo-realistic images, but this technology does not necessarily translate to generation of flawless comics. What is more, comic evaluation is a prominent challenge as common metrics such as Inception Score will not perform comparably, as they are designed to work on photos. In this paper, we: 1. Extend state of the art GANs to enable a comics-focused GAN architecture; 2) We implement DescriptionGAN, a novel text-to-comic pipeline based on a text-to-image GAN that synthesizes comics according to text descriptions. 3. We describe an in-depth empirical study of the technical difficulties of comic generation using GAN’s. DescriptionGAN has two novel features: (i) text description creation from labels via permutation and augmentation, and (ii) custom image encoding with Convolutional Neural Networks. We extensively evaluate the proposed DescriptionGAN in two
scenarios, namely image generation from descriptions, and image generation from dialogue. Our results on 1000 Dilbert comic panels and 6000 descriptions show synthetic comic panels from text inputs resemble original Dilbert panels. Novel methods for text description creation and custom image encoding brought improvements to Frechet Inception Distance, detail, and overall image quality over baseline algorithms. Generating illustrations from descriptions provided clear comics including characters and colours that were specified in the descriptions. ... Read more