The study of tumor microenvironments (TMEs) and immune cell composition in cancer, a disease characterized by uncontrolled growth and spread of tumor cells, has become increasingly important for understanding tumor progression and patient outcomes. Tools such as the TME-Analyzer enable this kind of research, but their manual workflows highlight a common problem in medical imaging: the scarcity of labeled data. This limits the efficiency and applicability of supervised learning algorithms to improve such medical image analysis tools. Self-supervised learning algorithms offer a promising alternative by learning feature representations without requiring labeled data. This thesis aims to address the issue of label scarcity by exploring the potential of self-supervised learning models for TME analysis involving the classification of individual cells in multiplex immunofluorescence (MxIF) microscopy images of triple-negative breast cancer (TNBC) tissue. To enable the learning of feature representations from MxIF images with an arbitrary number of color channels, this thesis proposes to pre-train an encoder network on every image channel separately according to the SimCLR algorithm and perform classification of multi-channel images by feeding the concatenated feature representation outputs of every channel to a classifier network — referred to as the Siamese configuration. A hyperparameter search is conducted to optimize the SimCLR encoder’s ability to learn high-quality feature representations of individual cells in MxIF images of TNBC tissue. Upon obtaining an optimal set of hyperparameters, the effectiveness of the learned feature representations in improving label-efficiency for individual cell classification is assessed. The results demonstrate that the proposed Siamese configuration improves the accuracy of classifying the inflammation status of TNBC tumor sections by 2.63%. Additionally, the optimal set of hyperparameters identified through the search include the use of the normalized temperature cross-entropy loss function with low temperature and an added image intensity thresholding term, as well as zoom and brightness/contrast augmentations. Furthermore, the optimized self-supervised learning model improves label-efficiency for individual cell classification, maintaining performance with only 40% of labeled data, while performance drops only when the label percentage is reduced below this threshold.

With the introduction of machine learning algorithms in industry came a new improvement to excising production line. Image recognisance can now be used to detect defects of the product made by these production lines. However these kinds of flaw detection can be fooled by mirror-like specular light effect on these products, which can reduce the reliability of these detection methods. In this thesis two approaches are taken to minimize these spurious specular light features. Here I look at images of welding spots taken from a car-door production line of Fiat Chrysler Automotive Italy. First a median filter is used on multiple images of welding spots, which each have a different illumination. Secondly the same images were fed into a modified algorithm developed by Antonello et al,2018 , originally used to split fore-and background of videos, to remove these spurious features. Then I tried to improve the output of this algorithm so that oversaturated pixels in the output were replaced by a combination of pixel values of the input images that were not oversaturated. Here is found that the median filter does not satisfy the desired goal of this thesis. The algorithm however gives a good filtered output of the input images and therefore are suitable to possibly be used for computer vision applications. The improvements made to the output of the algorithm do not correct the oversaturated pixels the right way, however the method I propose here seems to have some promising results if some improvement to this method are made, mainly concerning the normalisation of these pixels.

In the field of robotics, consider the following problem scenario: In a robot environment, a simple robot must push objects to reference places while figuring out which objects can be pushed, what the best manipulation strategy is, or which objects are static and cannot be pushed. The problem scenario can be decomposed into three research topics which individually have received much attention from the research community; learning object dynamics [8, 37], Navigation Among Movable Objects (NAMO) [7, 13, 21, 23] and nonprehensile pushing [2, 5, 25, 41, 42, 43]. A combination of these three topics could lead to improvements in planning, execution time, and reasoning, but it has not been explored in the literature. This thesis proposes a robot framework that combines these three research topics. This framework comprises of three key components: the hypothesis algorithm, the hypothesis graph, and the knowledge graph. The hypothesis algorithm computes a hypothesis on how to relocate an object to a new pose by computing possible action sequences given certain robot skills. In doing so, the hypothesis algorithm creates an hypothesis graph that encapsulates the structure of the action sequences and ensures the robot eventually halts. Once a hypothesis is carried out on the robot, information about the execution, such as the outcome, the prediction error, the type of controller used and other metrics, are stored in the knowledge graph. The knowledge graph is populated over time, allowing the robot to learn, for instance, object properties and then refine the hypothesis computed to increase task performance, such as success rate and execution time. A new planning algorithm is proposed that can detect a when a path is blocked by an object, the hypothesis algorithm relies on the newly proposed planner to generate action sequences and to free blocked paths. This planner extends the double tree optimised Rapidly-exploring Random Tree algorithm [7]. The planner constructs a configuration space for an object and is provided with starting- and target pose for that object. The planner then converts these poses to points in configuration space to then search for a path connecting the starting configuration to the target configuration. For the new planner, objects are initially classified as “unknown” and can later be categorized as either “movable” or “unmovable”. The object type information is then used when constructing the configuration space for the newly proposed planning algorithm. Configuration space consists of the conventional free- and unmovable- (or obstacle) space and the newly proposed unknown- and movable space. To carry out the investigation, a mobile robot in a robot environment with movable and unmovable objects is created. The robot is given a task that involves relocating a subset of the objects in the robot environment through driving and nonprehensile pushing. The task can be broken down into individual subtasks that consist of an object and a target pose. Planning for a push or drive action occurs with the newly proposed planning algorithm that, if successful, completes a given task and populates the knowledge graph with learned object information. Information that can be used to determine which objects to manipulate and what strategy performs best to manipulate a specific object. In an effort to develop a robot framework that combines these three topics, a framework is created that shows improved task execution as a result to experience gained in the robot environment. The proposed framework performs equivalent or better compared to the state-of-the-art frameworks that are specialized in only two out of three research topics [47]. It can be concluded that the framework partly combines the three topics because learning system models with a system identification module is moved to the future work section. Instead, the proposed method selects the best available control and system model combination in the set of available control and system model combinations.

Single Molecule Localization Microscopy (SMLM) has enabled researchers to breakthrough the diffraction limit and obtain nanometer resolution images of macromolecular structures. But due to the time involved in obtaining ample data for proper image, the technique is venerable to many problem including fluctuations due to thermal gradients from surrounding which cause the frames to drift. SMLM relies on the stochastic blinking of fluorophore probes. Thus drift in SMLM could be explicitly modelled as a stochastic state space process. These models could be used to perform drift correction. Two state space models are proposed relying on different properties of SMLM. The first model utilizes shifting of underlying image structure. The state space model for this property is constructed using shift matrices. A system identification method along with image reconstruction method is also derived to form the drift compensation algorithm for this model. This algorithm is further developed to provide adequate performance within low computational time. The second model relies on the position of emitter molecules and utilizes linking or pairing of fluorophore probes in succeeding frame to obtain the output data. Drift compensation algorithm for this model is constructed using Prediction Error Methods (PEM) and Kalman (RTS) smoother. The drift correction algorithm for these two models are also bench-marked with existing algorithms to obtain insight into performance. Furthermore, other properties of these algorithms are explored using simulation dataset and recommendation are provided for improvement and further research.

For a long time, scientists believed that it is not possible to achieve a better microscopic resolution than half the wavelength of light (~ 200 nm). During the last decades, various super-resolution fluorescence microscopy techniques have been developed, offering high-resolution imaging beyond the diffraction limit. It has been established as one of the major tools currently available for the study and analysis of biological tissues. The observation of, previously limited and unclear, cellular structures has demonstrated the great promise of super-resolution fluorescence microscopy in exemplifying biological processes at the cellular level. Inherently, single-molecule localization microscopy (SMLM) is based upon acquiring a sequence of diffraction-limited images, consisting of localizations of sparsely distributed emitting fluorophores. Next, precise and accurate estimates of the spatial coordinates of the fluorophores, placed on the structures of interest, are made. Subsequently, all recovered emitter positions from each frame are combined, and a super-resolution image is generated. One of the most crucial questions in SMLM concerns the precision with which the location of a single molecule can be determined. The Cramér–Rao lower bound (CRLB) is used to quantify the estimation performance in terms of localization precision. Recently, several modulation enhanced methods have shown resolution improvement by combining structured patterned illumination with SMLM techniques, such as MINFLUX and SIMFLUX. However, the analysis of images by SMLM and these modulation enhanced methods requires sparse activation regions to prevent overlap between active emitters. Recently, several methods have been developed with the ability to analyze higher density data, allowing partial overlap between emitters. In this work, we propose a maximum likelihood-based, modulation enhanced SMLM method able to localize multiple active emitters within a single region of interest (ROI), which we refer to as Multi- SIMFLUX. The mathematical framework of the algorithm is comprehensively built up throughout this work. The performance of the algorithm has been evaluated, and validated under a wide range of realistic imaging conditions. This involves various active-emitter densities, different emitter- and background intensities, numerous emitter configurations, a multitude of ROI-sizes and point spread function widths. As theoretically expected, our approach significantly outperforms conventional 2DGaussian SMLM with a twofold improvement factor in terms of localization precision for various emitter densities and intensities. At higher active emitter imaging densities, the extension from a single to a multi-emitter approach showed a clear improvement in recall fraction.

Currently used imaging methods in Atomic Force Microscopy (AFM) including the use of a Lock-In Amplifier or a Phase-Locked Loop, are suboptimal. In this report, the image resolution in AFM is improved by detecting the tip-sample interaction using complete measurements of the input of the cantilever and its measured deflection. Two methods are studied while assuming that the tip-sample interaction is sparse, namely a model-based approach and a data-driven approach. Real-life experiments have shown that the model-based approach improves the image resolution with a factor of 7.5 to 0.555 nm compared to the conventional imaging method, according to a metric using Fourier Ring Correlation in which a reference image is unnecessary. The data-driven approach can be used in the model-based approach to further improve the resolution. In addition to improved resolutions, a Linear Time-Invariant model of the mechanically driven AFM-cantilever immersed in liquid – from piezo input to cantilever deflection – has been obtained through subspace identification with a Variance Accounted For of 79.2%. Recommendations for future research include applying the latter model in detecting the tip-sample interaction, improving the data-driven approach, reducing the computational effort of the model-based approach and implementing algorithms for detecting the tip-sample interaction online.

Counter-acting image quality degradation caused by phase aberrations through physical correction requires the phase field to be known. As imaging hardware captures real-valued intensity, defined as the wave amplitude squared, obtaining this lost phase information is known as the phase retrieval problem and is a non-linear and non-convex optimisation problem. Literature treats this problem from two points of view: reconstruction using indirect phase sampling, and reconstruction using Fourier amplitude sampling. The former employs wavefront sensors such as the Shack-Hartmann sensor, which encodes phase gradient information in the form of the displacement of imaged spots. These methods, while fast, discard information such as interference and are limited to low-order reconstruction. The latter, also called wavefront sensorless phase retrieval, uses the full point-spread function (PSF) to obtain high-accuracy reconstruction at the cost of computational speed, number of intensity images required, or limited aberration magnitude. These two points of view have remained largely separated, but can be made compatible through a modelling technique called Shack-Hartmann diversity. This thesis explores the merging of wavefront sensorless methods with Shack-Hartmann intensity patterns to leverage more information from a single captured image frame. Firstly, a low-order modal reconstruction technique is presented applied to phase gradient fields obtained from Fourier demodulation of a Shack-Hartmann intensity pattern. A method of minimising the amount of redundant data-points used for reconstruction is illustrated through the removal of Fourier-interpolated data to speed up performance. Secondly, a novel extension of the Fourier demodulation technique to hexagonal Shack-Hartmann arrays is presented, allowing phase gradient extraction and modal reconstruction of hexagonal array intensity images using Fourier demodulation. Thirdly, Shack-Hartmann diversity is used to extend intensity-based modal phase retrieval using Taylor approximation of the intensity function to Shack-Hartmann intensity patterns. This bridges the gap between wavefront sensorless methods and Shack-Hartmann intensity patterns. Lastly, a novel hybrid method is presented for high-accuracy phase reconstruction based on applying the above intensity-based method to a single Shack-Hartmann intensity pattern with low-order pre-conditioning obtained from Fourier demodulation. The method is demonstrated to obtain highly accurate reconstruction on low-order aberrations, and better reconstruction accuracy on small-magnitude high-order aberrations with dominating large-magnitude low-order terms than traditional methods. Potential use cases are discussed, such as open-loop turbulence reconstruction for use in turbulence modelling.

For large-scale system with tens of thousands of states and outputs the computation in the conventional Kalman filter becomes time-consuming such that Kalman filtering in large-scale real-time application is practically infeasible. A possible mathematical framework to lift the curse of dimensionality is to lift the problem in higher dimensions with the use of tensors and then decompose it. The tensor-train decomposition is chosen due to its computational advantages for systems with low tensor-train rank. Within this thesis two main limitations of the existing tensor Kalman filter are solved. First, a method is developed based on tensor-train rank truncation of the covariances to increase the computational speed for more general systems. Second, a MIMO tensor Kalman filter is developed for a specific class of systems. The power of the developed methods is shown on the example of adaptive optics which fits into the framework. A comparison with state-of-the-art large-scale estimation algorithms shows the computational advantage of the tensor Kalman filter at the cost of approximation errors.

Significant work has been done in the field of computer vision focusing on learning and clustering methods. The use of improved learning methods has paved a way forward for researches to explore various theories to improve existing methods. One among various learning methods is Hierarchical learning which has showed impressive benefits and performance over traditional sequential learning approaches. In general, machine learning models require a lot of data for every new scenario which is not always possible and if so, is very expensive. Transfer learning, which focuses on transferring knowledge across trained machine learning models, is a promising machine learning methodology for solving the above problem. In this thesis, we propose an end-to-end neural network architecture on the NM500 neuromorphic chip using an incremental hierarchical learning approach. We first design a hierarchical representation of a taxonomy, develop a batch of pre-classifiers and use their output to construct a custom feature vector that is the input to the front-end network which learns the taxonomy. In other words, the taxonomy is embedded in the clustering method and not trained by a backpropagation algorithm. The custom feature vector has been structured to accurately incorporate the taxonomy based on the Manhattan distance norm. The structure has been proven mathematically and validated using experiments. A Radial Basis Function (RBF) is used for learning and a combination of RBF and K-Nearest Neighbors (KNN) for classification. The applicability of the proposed framework has been demonstrated on a road sign classification problem which is represented as a taxonomy. The ability of the framework to incrementally learn new categories and update the taxonomy online has also been shown. Lastly, we show a case of transfer learning where the entire back- end networks is used as a starting point to learn new features without significantly forgetting prior knowledge. This transfer learning framework showed comparable performance to the standard learning method in terms of accuracy while using significantly less labelled data. This work paves a way forward for researchers to develop transfer learning frameworks and more importantly explore neuromorphic hardware for machine learning tasks.

Mapping an environment with a Light Detection and Ranging (LiDAR) sensor through the use of a LiDAR Simultaneous Localization And Mapping (SLAM) algorithm is a powerful technology that allows for the creation of detailed 3D models. Recently various LiDAR sensors have been developed based on Micro-Electro-Mechanical System (MEMS) technology. These LiDARs are very low cost and considerably smaller than conventional LiDARs. They also often incorporate other sensors such as Inertial Measurement Unit (IMU)s and cameras into the same device. Performing LiDAR SLAM with MEMS based LiDAR is challenging due to the short range, the smaller Field of View (FOV) and the sensitivity to ambient light of MEMS based LiDAR. In this thesis the objective is to reduce the effect of these factors when doing LiDAR SLAM by incorporating IMU measurements into the position estimation of the sensor. A graph based positioning approach is proposed to achieve tight coupling of the IMU sensor and LiDAR position estimates. The method is made more robust by incorporating an outlier detection mechanism that reduces the influence of wrong LiDAR position estimates caused by insufficient points in the LiDAR FOV or by ambient light disturbance. The method was built in ROS and implemented on the Intel ® L515 sensor. The performance is evaluated in indoor situations with varying presence of ambient sunlight and where room size approaches the maximum limit of the sensor range. The algorithm achieves lower drift than the current state of the art for the Intel ® L515. The algorithm especially achieves altitude drift reduction and increases robustness to outliers in the LiDAR positioning.

Structured illumination microscopy (SIM) is a kind of wide-field super resolution imaging technique which achieves the resolution improvement by exploiting the interference patterns to obtain the information out of the observable region in Fourier space. The resolution improvement in SIM is done by analyzing the obtained image and restored the observed sample with software. This process is sensitive to the measured PSF, as the PSF is the main factor to do the image deblurring and parameters estimation. Frequently, the PSF of a SIM system is obtained by fitting the recorded beads image with the Gaussian function. However, the Gaussian fitting cannot retrieve the aberration and noise in the PSF. The blind image deconvolution method is a family of algorithms which can be used to restore the object and corresponding PSF from the degraded images. The novel Tangential Iterative Projections (TIP) algorithm is a kind of blind image deconvolution algorithm proposed by Wilding et al.. This algorithm is simple, fast and robust to the noise condition. It is proved that TIP can be implemented on both the multi-frame deconvolution scenario and single-frame scenario. The aim of this thesis is to verify whether TIP algorithm can retrieve a PSF, which can be used to do the SIM image reconstruction, from the SIM data. For this purpose, a simulated SIM optical system is built in Matlab to generate the SIM images, and both the single-frame TIP and multi-frame TIP frameworks are used to do the PSF retrieval. In addition, a new constraint for the PSF which is generated based on the diffraction limit of the optical system is implemented. In order to reduce the execution time and avoid the overfitting of the algorithm, a stop criterion is introduced. The result shows both the multi-frame TIP and single-frame TIP can be used to retrieve an effective PSF from the SIM data, and the single-frame TIP is more applicable and more accuracy compared with the multi-frame TIP.

Optical microscopes are fundamentally limited to a resolution of several hundreds of nanometers by the diffraction of light. Single-molecule localization microscopy (SMLM) circumvents this limit by sparsely exciting fluorescent molecules at different time instances. Single molecules can subsequently be localized with improved precision over the diffraction limit. Due to their high frame rate, single-photon avalanche diode (SPAD) arrays are imagers that can be used for SMLM. Most SPADs have to recharge after each photon arrival. During this recharging period, the SPAD is insensitive to more photon arrivals. As a result, SPAD arrays will measure zero or one photons for each pixel in each frame, whereas scientific complementary metal-oxide semiconductor (sCMOS) imagers and electron multiplying charge-coupled devices (EMCCD) have a discrete frame output. Here, we describe the photon arrivals in the image formation model of the SPAD array as a binomial process rather than as a Poissonian process. In addition, we quantify the minimum theoretical uncertainty of single-molecule localizations using a binomial Cramér-Rao lower bound and benchmark it with simulated and experimental data. We show that if the expected photon count is larger than one for all pixels within one standard deviation of a Gaussian point spread function, the binomial CRLB gives a 46% higher theoretical uncertainty than the Poissonian CRLB. Without saturation, which is the case for most SMLM applications, the binomial CRLB model gives the same uncertainty as the Poissonian CRLB. Therefore, the binomial CRLB can be used to predict and benchmark localization uncertainty for SMLM with SPAD arrays for all practical emitter intensities.