An increasing load on the Dutch rail network requires new solutions for railway monitoring as increasing wear and tighter schedules limit the possibilities of regular maintenance. UAV (Unmanned Aerial Vehicle) based photogrammetry is such a solution which limits the presence of people near the railway and train hindrance. Photogrammetric feature extraction methods struggle with extracting features from the rusty rail and reflective roll band. Therefore line features are used to aid reconstruction. Based on SfM (Structure-from-Motion) camera parameters, a line-based reconstruction is created of two data-sets. a A line detection algorithm is used to find line segments in the images. Found segments are matched between pairs of images by pixel window cross-correlation and geometric boundaries. Matched segments are then compared to find sets of three matching images. Reconstruction is done by minimizing line reprojection error in an iterative least squares solution. Results show promise for manually matched segments, however unfavourable automatic matching results prevent large scale reconstruction. A case study reveals matching struggles with camera rotation between images, matching problem mitigation leaves much room for improvement.