Print Email Facebook Twitter How Do Neural Networks Estimate Optical Flow A Neuropsychology-Inspired Study Title How Do Neural Networks Estimate Optical Flow A Neuropsychology-Inspired Study Author de Jong, D.B. (TU Delft Education AE) Paredes-Vallés, Federico (TU Delft Control & Simulation) de Croon, G.C.H.E. (TU Delft Control & Simulation) Date 2022 Abstract End-to-end trained convolutional neural networks have led to a breakthrough in optical flow estimation. The most recent advances focus on improving the optical flow estimation by improving the architecture and setting a new benchmark on the publicly available MPI-Sintel dataset. Instead, in this article, we investigate how deep neural networks estimate optical flow. A better understanding of how these networks function is important for (i) assessing their generalization capabilities to unseen inputs, and (ii) suggesting changes to improve their performance. For our investigation, we focus on FlowNetS, as it is the prototype of an encoder-decoder neural network for optical flow estimation. Furthermore, we use a filter identification method that has played a major role in uncovering the motion filters present in animal brains in neuropsychological research. The method shows that the filters in the deepest layer of FlowNetS are sensitive to a variety of motion patterns. Not only do we find translation filters, as demonstrated in animal brains, but thanks to the easier measurements in artificial neural networks, we even unveil dilation, rotation, and occlusion filters. Furthermore, we find similarities in the refinement part of the network and the perceptual filling-in process which occurs in the mammal primary visual cortex. Subject Biomedical optical imagingconvolutional neural networksEstimationGabor filtersneuropsychologyOptical computingOptical fiber networksOptical flowOptical imagingOptical sensorsVisualization To reference this document use: http://resolver.tudelft.nl/uuid:3ef12fc1-f8e0-473e-a7c8-06357e54061a DOI https://doi.org/10.1109/TPAMI.2021.3083538 Embargo date 2023-07-01 ISSN 0162-8828 Source IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 (11), 8290-8305 Bibliographical note Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public. Part of collection Institutional Repository Document type journal article Rights © 2022 D.B. de Jong, Federico Paredes-Vallés, G.C.H.E. de Croon Files PDF How_Do_Neural_Networks_Es ... _Study.pdf 3.83 MB Close viewer /islandora/object/uuid:3ef12fc1-f8e0-473e-a7c8-06357e54061a/datastream/OBJ/view