Automatic Quantification of Beach Occupation Using Oversegmentation and Machine Learning

Abstract

Decision- and policymakers responsible for the coastal zone aim at combining measures against for instance long-term erosion, with measures that have a positive social, economic impact on the region. Recreational beach usage has a large social economic impact on a region and therefore quantification of the recreational beach usage can provide information on the social, economic situation in a region. In the Netherlands beach usage quantification is mostly performed by manual counting during a limited number of days in the field and this limits the spatial and temporal resolution. The objective of this study is to develop and test a method for accurate, robust and automatic monitoring of the spatial and temporal distribution of the number of beach users on the Dutch coast.
Multiple approaches to quantify the number of persons in an area are reviewed. Comparison of the reviewed approaches showed that a first distinction can be made between methods using a visual light (camera-) sensor and other methods based on the use of Bluetooth, Wifi, GPS-location (all related to phones) and LiDAR. Based on a literature review the visual light sensor is decided upon to best suit beach user quantification. Within the visual light approach a second distinction is made between methods based on the difference in pixel intensity, a method based on variance of pixel intensities over multiple frames and a method using oversegmentation combined with a machine learning framework for classification.
The difference in pixel intensity method is observed most often in literature, but has limitations in conditions that are concerned typical for the Dutch coast (e.g. clouds). The method based on the variance of pixel intensities and the method using oversegmentation are possibilities to overcome the problems described in the studies on the pixel intensity method. Based on preliminary tests the oversegmented machine learning approach is selected because it does not require beach users to move to be detected. Moreover, single snapshots can be evaluated which require a limited data infrastructure in-situ and this is considered advantageous regarding the ease of implementation and cost effectiveness.
The oversegmented machine learning approach divides images into small regions of similar pixels based on pixel gradients. The regions are called superpixels and superpixels can be characterised by significantly more features than the conventional r,g,b relations corresponding to regular pixels evaluated in the differences in pixel intensity method. The availability of an increased number of features provides more options to distinguish between classes during classification and this can be advantageous in difficult (e.g. cloudy) conditions. Classified beach user superpixels can represent multiple beach users due to for instance occlusion and therefore a regression relation between classified beach user superpixels and a manually counted ground truth is determined for conversion of classified beach user superpixels to the number of beach users. Hence, the oversegmented machine learning method for quantification of beach occupation combines an oversegmented classification model to classify superpixels into classes (e.g. beach user and sand) and a regression model to convert classified beach user superpixels to beach users. 
The oversegmented machine learning method has previously been evaluated in the study of Hoonhout et al. (2015) for the classification of coastal images into the classes 'water', 'sand', 'objects', 'vegetation' and 'sky' and this led to the open-source toolbox Flamingo (Hoonhout and Radermacher, 2014b). The current study adapts and develops the Flamingo toolbox for the quantification of beach occupation. The impact of changing parameters of the existing toolbox on the oversegmented classification model are evaluated to obtain insight in the parameters that have to be changed to apply the toolbox to the quantification of beach occupation. The influence of the parameters: class aggregation, measures to take account for imbalances in the dataset, regularisation, number of images in the training dataset, image enhancement, addition of articial channels to enable more (new) features and the required number of features are reviewed. Especially changes in the parameters class aggregation, number of images in the training dataset and articial channels affect the overall model performance. The effect on the overall model performance of measures to account for imbalances in the dataset is limited. However, these measures can change the relative distribution of precision and recall corresponding to the false negative and positive rates respectively. The final classification model is trained and validated with a dataset containing 76 manually annotated images, default undersampling to account for the imbalance in the dataset and added articial channels. A 4-class model with classes beach users, sand water and objects proved to be the best performing class aggregation.
The classified beach user superpixels are converted into a number of beach users with a regression model obtained by fitting a second order polynomial regression line to the classified beach user superpixels of the training images and the corresponding manually counted ground truth. Evaluation of the fit shows that the oversegmented machine learning method is a suitable method for quantification of beach occupation indicated by a R2 of 0.92. The regression model is validated by application of the combined  oversegmented classification- and regression models on a new and 'unseen' dataset of 80 images. Validation shows that the regression model is applicable on images that are not used during development of the model (R2=0.87) and this moreover confirms the suitability of the oversegmented machine learning method. Analysis of the largest errors showed that especially unoccupied beach stretchers and images captured by an unclean lens limit the performance of the oversegmented machine learning method.
The developed oversegmented machine learning method is benchmarked against one of the differences in pixel intensity methods representing the current state-of-the-art. The benchmark shows that the oversegmented machine learning method (R2=0.87) has a higher performance on the evaluated evalidation dataset compared to the method representing the current state-of-the-art (R2=0.76). The difference in performance indicates that the newly developed method is more suitable to the varying conditions associated with the Dutch coast.
Tests of the oversegmented machine learning model on a different camera station than was used for training of the oversegmented classification model, did not lead to satisfactory results. This indicates that the current approach for application on camera stations not used during training is not suitable. Therefore, at this point, the oversegmented machine learning method lacks robustness with respect to the performance on multiple different camera stations. A number of possible causes for the limitedperformance are treated and provide recommendations for further research. 
The presented oversegmented machine learning method, despite its limitations, provides an opportunity to quantify beach occupation with a high temporal and spatial resolution in variable (weather) conditions that are known to limit the performance of the current state-of-the-art methods and are typical for the Dutch coast. The method, therefore, enables the possibility to monitor locations in conditions that with the current state-of-the-art would be diffcult to monitor.