This dissertation investigates the use of deep-learning methods for user-guided content creation, manipulation, and exploration. It illustrates the potential of neural techniques to support working with large data collections and we illustrate our solutions through several applications. Regarding content generation, we propose an algorithm to produce material representations from a single image. We illustrate content manipulation with an approach to perform perceptually plausible interpolation and examine exploration in the context of interactive retrieval. For the latter, we show that features spaces are of high relevance to organize data and show the generality of this concept by proposing novel exploration methods for image and music collections.

Ray tracing has experienced increasing adoption in various spaces of computer graphics. The ReSTIR (Reservoir-based Spatiotemporal Importance Resampling) family of techniques has enabled several orders of magnitude speedups in light transport simulation algorithms which rely on ray tracing.

We introduce an extension to WRS (Weighted Reservoir Sampling), a key component of ReSTIR, to reduce the occurrence of duplicate samples in multi-sample reservoirs. Further, we show how samples from multiple reservoirs can be combined in an MIS-style estimator as opposed to resampling from them. Lastly, we combine these two components to compute optimal weights for this estimator in a similar vein to OMIS (Optimal Multiple Importance Sampling).

Our direct lighting proof of concept implementation demonstrates the efficacy of our WRS extension, lowering the variance of ReSTIR, particularly with difficult lighting arrangements. Further, our MIS-style estimator shows a significant improvement compared to ReSTIR. In problem domains where resampling produces a function value for integration, such as path tracing, this comes at no additional cost. Lastly however, optimal weights do not appear to be beneficial for our approach.

The prevalence of partial differential equations (PDEs) in modeling physics and the low speeds of numerical solvers demands more efficient solving methods. For this purpose, machine learning based methods have been proposed, but these are typically discrete, difficult to interpret and require large amounts of ground truth data to train. To address these issues, we propose a novel machine learning based solver that builds upon advancements in Gaussian based reconstruction. Our method represents the solution to a time-dependent target PDE purely in terms of Gaussians, making it completely continuous and meshless. These Gaussians are evolved by means of an autoregressive neural network that is applied to each Gaussian, integrating information from a local neighborhood of Gaussians. This enables the change of position, scale, rotation and value of the Gaussians to model the solution in a particle-like manner. Extensive experiments show the potential for Gaussians to model arbitrary physical phenomena. We also compare our approach to ground truth data and various state of the art methods, which demonstrates that the method performs well on short-term prediction, but does not match the state of the art for long-term prediction.

The bilateral filter is a popular filter in image processing and computer vision. This comes from the fact that it is able to blur images while keeping the structure intact. However, the bilateral filter allows for blurring to happen across edges. This can result in halo-like effects around the edges of structures if both sides are made up of different intensities. In this paper, we propose an extension to the bilateral filter that reduces this phenomenon of blurring across edges. By giving the filter knowledge of the edges beforehand, it is possible to prevent the filter from blurring past them. When we filter a pixel, its surrounding area within the kernel is checked for edges. If a pixel within this area lies on or beyond an edge, its weight for blurring is reduced. As a consequence, pixels that lie past an edge have less influence on blurring. We show that this new edge-aware bilateral filter reduces across-edge blurring compared to the standard bilateral filter. Furthermore, when we allow a bigger range of intensities to mix, the new filter is also able to prevent the filtered image from appearing washed out, unlike the bilateral filter.

This paper introduces the Quadrilateral filter, an advanced extension of the Bilateral and Trilateral filters aimed at addressing limitations in high-gradient regions of images. While the Bilateral filter effectively preserves edges during smoothing, it struggles with intensity variations, leading to blunted image details. The trilateral filter improves upon this by incorporating local plane geometry approximations but assumes linear pixel intensity distributions, limiting its effectiveness. The proposed Quadrilateral filter utilizes curvature-based geometry approximations to enhance noise reduction, contrast preservation, artifact reduction, and image reconstruction by accounting for nonlinear pixel value distributions. The development of this filter represents the main contribution of the paper while exploring whether the established Bilateral and Trilateral filters’ performance can be further improved through curvature-based local geometry approximations. The findings demonstrate improvements in image quality and detail preservation, with broad implications for applications in image de-noising, tone-mapping, multimedia processing, and beyond.

Unlike traditional blur filters, the bilateral filter exhibits non-linear blur behaviour as its kernel size increases. This atypical blur behaviour makes it challenging to find a good σr . This paper investigates the underlying reasons for this behaviour and proposes methods to align the bilateral filter’s blur scaling linearly with its spatial filter size. Using local frequency analyses to quantify blur levels, we introduce an approach that finds the best σr through iterative search. Results demonstrate that the pro- posed method effectively counters the atypical blur behaviour. However, the proposed method does not perform sufficiently when handling very large kernel sizes. The proposed method can be used to abstract away the σr parameter when seeking linear blur behaviour in the bilateral filter. Further re- search is needed to make it functional for very large kernel sizes.

Traditional bilateral filters, effective in 2D image processing, often fail to account for the 3D structure of meshes, leading to artifacts in texture filtering. This thesis introduces On-Mesh Bilateral Filtering, a novel method that adapts the bilateral filter to work with non-contiguous texture mappings by incorporating 3D spatial distances and face adjacency information into the filtering process. The On-Mesh Bilateral Filter combines mesh surface sampling techniques with heat geodesic distance calculations to create a geometry-aware kernel that achieves more accurate and context-sensitive smoothing operations, respecting both the mesh topology and texture space properties. This paper hopes to encourage further research in the area of geometry-aware texture filters.

The bilateral filter is an edge-aware image filter. While it has a variety of applications, its naive implementation is quadratic in nature, hindering the ability to efficiently process multi-megapixel images. If performance is needed, like in a real-time setting, an approximation is necessary. Current literature on Fourier series-based approximations does not explore the capabilities of graphics processing units (GPUs) as viable platforms for this computational problem. This paper proposes an approach for implementing such filtering on a GPU by conducting a series of separable convolutions, and also investigates the use of different range kernels. Our adaption of the bilateral filter is found to be more two times faster than readily available solutions, with frame times showing that real-time performance is possible for large spatial kernel sizes and image resolutions.

Dimensionality reduction is an important task in high-dimensional data visualisation. Among the popular algorithms for achieving this is t-SNE, which aims to preserve local neighbourhoods in the lower-dimensional embeddings. While t-SNE traditionally works in Euclidean space, embedding in hyperbolic space offers several advantages, specifically for data of arbitrary size and exponential growth, such as tree-based structures. We propose a new solution to approximate and accelerate the calculation of t-SNE gradients using a uniform grid structure. This new method produces embeddings with better neighbourhood preservation than previous solutions, while also providing better runtime performance.

In this work we aim to implement a variaton of the acceleration of hyperbolic t-SNE done by Skrodzki et. al. [19]. This variation aims to embed the points in the Klein Disk model of hyperbolic space instead of the Poincar ́e Disk model using an altared version of a polar quadtree to speed up the computation in a similar fashion as the Barnes-Hut scheme for the Euclidean versino of t-SNE. We analyze our results to prove our acceleration works for the Klein Disk model and compare the efficiency of our implemen- tation to the one for the Poincar ́e model in terms of quality of results and runtime.

This paper investigates a method for accelerating hyperbolic t-SNE — a popular high-dimensional data visualization technique. In particular, it focuses on building a hyperbolic t-SNE variant that uses a different model of hyperbolic space (called the Lorentz Hyperboloid model) for representing the low-dimensional embeddings. An acceleration algorithm based on a tree data-structure is then used to achieve a better asymptotic runtime complexity compared to the original version. The paper then compares this implementation with other alternatives — including accelerated variants — and shows that it computes embeddings of better quality at a similar rate.

With the rapid growth in data collection, efficient data processing is critical. Dimensionality reduction methods, like t-distributed stochastic neighbour embedding (t-SNE), compress high-dimensional data into embeddings that preserve the key features of the datasets making data less sparse and easier to process further. Recent improvements suggest that using hyperbolic space for data representation can benefit embeddings. As it is a new technique, it remains computationally expensive. Previous work suggests that a Barnes-Hut approximation with a polar quadtree can be applied to the Poincaré disk model to approximate the result of hyperbolic t-SNE and accelerate its calculation. However, the polar quadtree is proposed as a solution to accelerate the calculation without exploring alternative approaches. Aiming to close this gap, we propose an acceleration method using Barnes-Hut approximation with a Cartesian quadtree. We experimentally compare our acceleration method to a polar quadtree and showcase its lower execution time without the loss of quality of the embeddings.
Implementation and scripts for the experiments and plots are available at https://github.com/Sne4kers/hyperbolic-tsne

The objective of this project is to train a model that transforms a tree with its foliage into only its branch structure. This is achieved by employing machine-learning techniques, specifically Generative Adverserial Networks (GANs). By utilizing the proposed method, a predictive model is built that automatically minimizes its own error function through a comparison of a set of input and ground-truth tree images, which are tree images with and without leaves, respectively. The adoption of GANs has shown promising results, both visually and metrically.

The most established and widely used methods for analysing tree images for tasks such as geometry analysis, segmentation and classification often rely on pixels. In this paper, the applicability of analyzing tree geometry based on a graph representation rather than a pixel-based approach is pursued. To do so, 2D renders of different species of trees are converted to spatial graph structures capturing significant points on the tree skeleton. Two independent Graph Convolutional Network algorithms which learn node (coordinate) features are then applied on the obtained dataset to assess the reliability of graph based analysis. The first experiment explores a GCN for assigning correct species labels to the skeleton graph of the original tree image, demonstrating the association between geometry and tree metadata. The second experiment, an unsupervised representation learning, is conducted by using Graph Autoencoders to obtain an embedding for each skeleton graph which can be used to reconstruct partially the same graph, demonstrating the association between GCE latent representation and geometry. Promising results were found in both cases, reinforcing the reliability of the original proposition to rely on geometry as well as pixels for tree analysis tasks.

L-Systems allow for the efficient procedeural generation of trees to be used for rendering in video games and simulations. Currently, however, it is difficult to engineer grammars that mimic the behaviours of real life trees in 3 dimensions. To be able to deduce them, the skeleton of a tree can be used to train a model and generate an L-system for a given tree in particular. The aim of this paper is to provide a pipeline to isolate these skeletons from images of a tree, using Neural Radiance Fields (NeRFs) to reconstruct the tree, and using Laplacian Based Contraction to retrieve the underlying skeleton. We find that this approach leads to 3-dimensional topologies that very closely resemble the given tree.

Trees are essential components of both real and digital environments. Therefore, it is important to have 3D models of trees that are of high quality and computationally efficient. One way to achieve this is by compressing a high-quality model using billboard rendering, which involves partitioning the tree into multiple planes to produce a similar result to the original. Our study explores the compression of 3D models using an optimization loop and adapting billboard rendering techniques. We use computer vision primitives to render basic models, which we then optimize by adjusting the texture to resemble the original tree. The models consist of multiple upright planes that are rotated around the central vertical axis of the original tree. We use different optimization functions, such as L1 and L2 losses, to determine the best approach. We can improve the initial models by bounding the billboards and limiting their heights and widths to that of the trees. Additionally, we can use double-sided textures for the billboards to allow more flexibility for optimizing different species of trees. However, optimizing multiple tree types performs differently for each species, leading to improvements that only benefit certain trees in specific scenarios. Using quantitative metrics, we determined which models perform best and how similar they are to the original after training. We found that our compressed models generally resemble the original while having only a fraction of the original size.

Low-dimensional datasets, for which each datapoint contains no more than three attributes, are straightforward to visualize with common visualization idioms, such as scatterplots. In order to visualize high-dimensional datasets with potentially thousands of attributes, their dimensionality will need to be reduced. t-SNE is a widely used, state-of-the-art algorithm for non-linear dimensionality reduction. It produces embeddings of the high-dimensional data onto two or three dimensions by using gradient descent to minimize the discrepancy between the probability distributions of the high-dimensional and low-dimensional datapoint similarities, iteratively adjusting the low-dimensional embedding. However, since it is impossible to capture all neighbourhoods over all high-dimensional axes in two or three dimensions, the resulting clustering is merely virtual, only meant to offer an intuition of the dataset's high-dimensional distribution. Our work proposes a number of tools that allow a user to interactively impose constraints based on prior knowledge or assumptions about the dataset and manipulate the underlying probability model with the aim of enhancing an embedding's interpretability, for a better intuition about the data's high-dimensional structure. We demonstrate the effectiveness and limitations of these tools with examples and case studies.

Image inpainting is a problem that has been well studied over the last decades. In contrast, for 3D reconstructions such as neural radiance fields (NeRFs), work in this area is still limited. Most existing 3D inpainting methods follow a similar approach: they perform image inpainting on the training images and use the inpainted images for further training of the 3D model. Due to inconsistencies in the different inpaintings of the images, the 3D inpainting often becomes blurry. With the advent of 3D Gaussian Splatting (3DGS), we identify a new opportunity for 3D inpainting. As 3DGS is more explicit in nature than NeRF, we can manipulate the 3D Gaussians directly rather than relying on image inpainting. Based on that key idea, we propose a method that works similar to the PatchMatch image inpainting algorithm. We first construct a nearest-neighbour field (NNF) by searching for nearest-neighbour patches throughout the scene that look similar to the area we want to inpaint. After constructing the NNF we copy the contents of the nearest-neighbour patches to the inpainting region and blend them together to obtain the inpainting result. In our experiments we found that our method performs well in terms of texture synthesis but struggles with structure synthesis, similar to the original PatchMatch algorithm. In cases where only texture synthesis is required to inpaint the area our method is able to provide good results, although in some cases pre-processing of the scene is necessary, as we found that better quality inputs (e.g. the scene itself, the surface mesh underlying the scene, and precise masks) drastically improve the results of our method. Moreover, some parameters of the algorithm are highly scene-dependent and by tailoring them to the scene we can further enhance the performance of the algorithm. Besides introducing a 3D inpainting method that directly manipulates the scene contents, our work offers valuable new insights into 3DGS editing in general.

This dissertation develops intrinsic approaches to learning and computing on curved surfaces. Specifically, we work on three tasks: analyzing 3D shapes using convolutional neural networks (CNNs), solving linear systems on curved surfaces, and recovering appearance properties from curved surfaces using multi-view capture. We argue that we can find more efficient and better performing algorithms for these tasks by using intrinsic geometry.

Chapter two and three consider CNNs on curved surfaces. We would like to find patterns with meaningful directional information, such as edges or corners.
On images, it is straightforward to define a convolution operator that encodes directional information, as the pixel grid provides a global reference for directions. Such a global coordinate system is not available for curved surfaces. Chapter two presents Harmonic Surface Networks. We apply a 2D kernel to the surface by using local coordinate systems. These local coordinate systems could be rotated in any direction around the normal, which is a problem for consistent pattern recognition. We overcome this ambiguity by computing complex-valued, rotation-equivariant features and transporting these features between coordinate systems with parallel transport along shortest geodesics.

Chapter three presents DeltaConv. DeltaConv is a convolution operator based on geometric operators from vector calculus, such as the Laplacian. A benefit of the Laplacian is that it is invariant to local coordinate systems. This solves the problem of a missing global coordinate system. However, the Laplacian operator is also isotropic. That means it cannot pick up on directional information. DeltaConv constructs anisotropic operators by splitting the Laplacian into gradient and divergence and applying a non-linearity in between. The resulting convolution operators are demonstrated on learning tasks for point clouds and achieve state-of-the-art results with a relatively simple architecture.

Chapter four considers solving linear systems on curved surfaces. This is relevant for many applications in geometry processing: smoothing data, simulating or animating 3D shapes, or machine learning on surfaces. A common way to solve large systems on grid-based data is a multigrid method. Multigrid methods require a hierarchy of grids and the operators that map between the levels in the hierarchy. We show that these components can be defined for curved surfaces with irregularly spaced samples using a hierarchy of graph Voronoi diagrams. The resulting approach, Gravo Multigrid, achieves solving times comparable to the state-of-the-art, while taking an order of magnitude less time for pre-processing: from minutes to seconds for meshes with over a million vertices.

Chapter five demonstrates the use of intrinsic geometry in the setting of appearance modeling, specifically capturing spatially-varying bidirectional reflectance distribution functions (SVBRDF). A low-cost setup to recover SVBRDFs is to capture photographs from multiple viewpoints. A challenge here, is that some reflectance behavior only shows up under certain viewing positions and lighting conditions, which means that we might not be able to tell one material type from another. We frame this as a question of (un)certainty: how certain are we, based on the input data? We build on previous work that shows that the reflection function can be modeled as a convolution of the BRDF with the incoming light. We propose improvements to the convolution model and develop algorithms for uncertainty analysis fully contained in the frequency domain. The result is a fast and uncertainty-aware SVBRDF recovery on curved surfaces.