The accuracy and comprehensiveness of 3D city models have become increasingly important for applications like monitoring, sustainability evaluation, disaster management, and urban planning. However, creating accurate and complete 3D models is challenging. Traditional methods, such as photogrammetry and Light Detection and Ranging (LiDAR), often face issues like time-consuming processes and data gaps caused by occlusion. Additionally, these methods typically produce discrete models that fail to capture all the information from the original objects, limiting the ability to reconstruct small structures.

In this thesis, we explore the potential and characteristics of enhancing 3D models for urban areas using implicit neural representation. This method offers generalizability, ensures no void areas, and provides more detailed information when using multi-modal inputs. The datasets used for model training include Actueel Hoogtebestand Nederland 3 (AHN3), 2019 Luchtfoto Beeldmateriaal, 3D Basisregistratie Adressen en Gebouwen (BAG), and Basisregistratie Grootschalige Topografie (BGT). The city center of Eindhoven is used for training and testing, and the city center of Rotterdam serves as a test dataset.

The method involves learning location-dependent latent codes from raw point cloud and orthophoto, and adjusting the probability of space occupancy based on point clouds sampled from the 3D city model. A decoder is used to calculate the probability of existence for any point in the 3D space from the continuous field. Proper sampling allows us to derive a continuous Digital Surface Model (DSM) with unlimited resolution from the neural network.

The results show a high degree of accuracy: the generated DSM for the training area in Eindhoven demonstrated a median absolute error of 0.484 m overall. Accuracy for building areas was recorded at 0.798 m, and for terrain, it was 0.302 m. The model also displayed robust generalization capabilities, with an accuracy of 0.336 m in the Eindhoven test area and 0.23 meters in Rotterdam. Additionally, the model's ability to fill voids was confirmed through both visual inspection and quantitative evaluations. This research underscores the potential of implicit neural representation for generating detailed DSMs and effectively filling no-data voids.

Multiview Stereo (MVS) reconstruction techniques have made significant advancements with the development of deep learning. However, their performance often deteriorates in low-light conditions, where feature extraction and matching become challenging. Traditional image enhancement solutions are insufficient for MVS tasks in low illumination, relying on manual adjustments. We introduce an end-to-end MVS framework incorporating a diffusion-based image enhancement algorithm with MVS to build an end-to-end framework for improving the performance of MVS in low-light conditions. This integration improves color rendering and visualization of 3D reconstructions and slightly enhances geometric shapes. Our method uses a feature adapter to integrate the enhanced images from the Low-light Diffusion model into CasMVSNet, refining the feature maps in poorly lit environments. Validation on the DTU and Tanks and Temples datasets demonstrates our model’s robustness and generalizability across various lighting conditions and MVS pipelines, including GeoMVSNet and MVSNet. Our approach simplifies the training process by requiring only the training of an adapter rather than a multi-view image enhancement model, underscoring the effectiveness of incorporating image enhancement into learning-based MVS frameworks for low-light conditions.

In this thesis we present a new idea to objectively assess reconstruction algorithms. Because it is not feasible to completely scan a high-detailed ground truth mesh of large urban objects, the performance of the reconstructed meshes can therefore not be measured objectively. To solve this, we present a new mesh evaluation methodology that can more objectively assess the quality of the generated mesh based on a low detailed ground truth mesh. We achieved this by creating a synthetic dataset based on low-detailed models of large urban buildings and using this as a ground truth mesh and input data for the reconstruction algorithms. Thanks to our new methodology, we were able to compare the output mesh and ground truth mesh using a wireframe model of the meshes. This allows us to give a more objective score to the results without having to look at entire model, which is the usual method. The results of this thesis show that the new methodology has potential to be used for creating a new benchmark, and it opens a new door to using more readily available objects that could not be used before.

Generative AI is developing rapidly and has shown great potential in generating and editing images with text prompts. It has achieved partial success in the challenging 3D model edits of common objects. However, there is a lack of attention in the building domain, which already faces certain limits in the 2D space. 3D building models automatic edit has wide application potential, especially for renovation, concept comparison, and large-scale scenes. Therefore, the thesis aims to edit the building models effectively through texture creation and geometric changes with generative AI by evaluating the existing 3D edit pipelines and putting forward practical solutions based on them.

The thesis first examines the current methods. Latent-paint, Text2Tex and X-Mesh are chosen as the representative existing explicit representation (mesh) based pipelines for their relatively satisfying performance in common objects and the different ideas contained. Modifications are made based on them to improve output quality from the aspects of the control module and edit module. For the first aspect, attempts include adding the view specification text prompt to the original X-Mesh and using the image as the control for X-Mesh. Image control X-Mesh is also further experimented with placing higher attention on the input image view and editing the key semantic parts of the building separately. For the latter aspect, modifications include freezing sampled vertices geometry and combining X-Mesh and Text2Tex.

Results show that Latent-Paint mainly creates texture with only major colour and lacks details. Text2Tex generally generates fine textures for inputs with richer depth information. X-Mesh creates textures and edits geometry jointly. It can improve the fidelity of the input mesh to some extent but suffers from noise problems. Image control X-Mesh generates more realistic results and outperforms the text control pipeline. Combining Text2Tex with X-Mesh takes advantage of Text2Tex, which can generate smoother and more realistic textures compared to X-Mesh in some cases. The combination is especially recommended for low-fidelity mesh of small or medium size. The user study also shows that the combination of X-Mesh and Text2Tex generates the most favoured results while image control X-Mesh ranks second. There are still limitations in geometric deformation scope, fidelity, generalizability and computational efficiency in the proposed 3D edit pipelines. The thesis succeeds in proving the application potential of 3D edit pipelines in the building domain and obtaining high-quality results by modifying existing methods.

Urban Energy Modeling (UEM) provides a comprehensive approach to urban planning, helping to create sustainable, resilient, and energy-efficient cities that meet the needs of current and future generations. The key inputs for UEM methodologies and tools are the geometry of the building stock and its thermophysical properties. In the Netherlands, the 3DBAG provides the building stock geometry, while the thermophysical properties can be approximated using energy consumption estimates specific to each residential building type from the IEE project TABULA. However, a challenge arises as open data on residential building types at a national level is currently not readily available, necessitating the development of a method to infer this information from other accessible data sources.

In response to similar successful studies, this thesis also focuses on utilising machine learning to address this challenge. Support Vector Machine (SVM) and Random Forest (RF) algorithms were tested and compared. These algorithms were trained using data on the residential building types obtained from the Rijssen-Holten energy testbed and EP-online, with the latter requiring preprocessing to obtain the relevant information. Additionally, 25 features derived from cadastral data and building geometry underwent a selection process to identify the essential features for accurate classification of building types.

Eight models were trained and applied across eight case studies, containing subsets of the Netherlands representative for the whole country. The combined results were analyzed to determine necessary features, required data, and the most suitable machine learning approach for this research.

The results revealed that features such as adjacency to other buildings, width, and volume in LoD2.2 correlated the most with Dutch residential building types. However, critical features like the number of storeys, the presence of an open porch, or galleries were not available as open data, even though they directly relate to the definition of certain residential building types.

This thesis presented successful models, demonstrating accuracies between 61.1% and 98.5%, and balanced accuracies ranging from 51.6% to 94.2%. Importantly, performance differences were observed in various case studies, particularly in distinguishing accuracy between multi-dwelling and single-dwelling houses. Despite the longer tuning and training time, the suitability and accuracy of the RF models generally outperformed the SVM models.

These findings highlights the capacity of machine learning to attain robust classification outcomes for the Dutch building stock when trained on representative datasets. Nevertheless, it is emphasized that the accuracy of results is contingent on data quality, and difficulties may arise in scenarios involving intricate buildings with multiple components and ambiguous classification rules.

Style transfer is a recent field in the development of deep neural networks, which allows for the style from one image to be transferred onto another image. This has been well-researched for 2D images, but transferring style onto 3D reconstructed content can still be further developed. Being able to style a 3D reconstruction would allow users to recreate anything in the real world, such as a chair, with any style they see fit. Where other methods use texture-based approaches which often create low quality geometry and appearance, or
use radiance fields which style a whole scene instead of just the 3D reconstructed object, we have developed a method which styles an implicit surface.
We achieve this by using Implicit Differentiable Renderer (IDR), which trains, using masked
images as input, two neural networks that learn the geometry and appearance. Rendered views of the object are styled using 2D neural style transfer (NST) methods, and the style information is used to further train the appearance network to display the given style. With Masked deferred back-propagation we are able to optimize the appearance renderer, which is normally trained on only patches of the rendered image to save memory, while using style
transfers designed for full-resolution images. We showcase different results from our method using different 3D reconstruction datasets and style images, and showcase how to implement a user-created dataset. We carry out
extensive tests on what effects different parameters have on the final result. Comparing our results to similar 3D style methods demonstrates that our method performs equally well in achieving faithful style transfer, while having the benefits of creating high quality geometry and only styling the reconstructed surface.

Piping and Instrumentation Diagrams (P&IDs) are graphical representations utilized in chemical engineering plants. Due to confidentiality reasons and legacy drawings, these diagrams are sent in PDF format. Piping engineers need to make a material take-off (MTO), a document containing all the components of a P&ID from these drawings. Today, this is done manually, which proves to be time-consuming and laborious. A piping engineer spends approximately 36 hours per 10 P&IDs, with an average of 500-1000 P&IDs per project. Given the expertise and value of process engineers, this manual counting process incurs substantial costs and a repetitive workload. Consequently, there is a growing motivation to automate this process.

In response, this thesis introduces an innovative deep learning model, PandID-Net, designed specifically for P&IDs. PandID-Net uniquely integrates symbol detection, line detection, and text recognition into a single model, diverging from previous methods that relied on separate models and rule-based techniques. It is the first method that uses deep learning for the line detection task in P&IDs. This all in one approach not only simplifies the processing pipeline but also enhances computational efficiency in detecting and pinpointing symbols, lines, and text, as well as their interrelationships.

The optimal configuration of PandID-Net is found by an ablation study where the performance of individual components is tested in isolation. This optimized configuration is then evaluated and benchmarked against a prior study by Paliwal et al. on the same dataset. PandID-Net achieves a performance in F1 scores of 92.89 and 94.48 for line detection and keypoint detection respectively

This thesis introduces a Learned-Based Multi-View Semantic Stereo method, addressing the limitations of traditional and learned-based Multi-View Stereo (MVS) techniques in reconstructing reflective and low-textured regions, particularly prevalent in 3D models of buildings. Traditional methods lack completeness, while learned-based methods struggle with accuracy. Focusing on enhancing 3D models of buildings, this research integrates semantic information into the existing deep learning architecture for depth prediction, specifically CasMVSNet, to guide the reconstruction process. Three key strategies are employed: first, the incorporation of semantic maps into the network through a multi-modal approach; second, the introduction of a multi-modal refinement module at the end of the CasMVSNet model to improve the initial output depth maps; and third, the introduction of two new loss terms designed to enforce varying degrees of smoothness on specific semantic categories. Experimental results, conducted on the DTU dataset, demonstrate a significant enhancement in accuracy at the point cloud level while maintaining the completeness of the reconstructed models. Validation and generalization on the ETH3D dataset show consistent patterns. This research showcases the potential of integrating semantic guidance in 3D reconstruction of buildings, advancing the field of computer vision.