Once data have been recorded, data validation procedures have to be conducted to assess the quality of the data, i.e. give a confidence grade. Furthermore, gaps may occur in time series and, depending on the purposes, these can be given values by application of e.g. interpolation. Since both aspects are strongly correlated, this chapter gives an overview on the main data validation and data curation/imputation methods. Instead of offering exhaustive details on existing methods, this chapter aims at providing concepts for most popular techniques, a discussion of their advantages and disadvantages in the light of different cases of application, and some thoughts on potential impacts of the choices that must be made. Despite involving mathematical methods, data validation remains a largely subjective process: every data user must be aware of those subjectivities.

This introductory chapter indicates why well-defined, high-standard, and reliable monitoring is a key aspect in the necessary evolution of urban drainage and stormwater management and why it should become routine practice. It provides a framework, guidelines, and recommendations to define monitoring objectives and means. It also presents the structure and the chapters of the rest of the book.

The knowledge of water levels and discharges in urban drainage and stormwater management (UDSM) systems is of key importance to understand their functioning and processes, to evaluate their performance, and to provide data for modelling. In this chapter, devoted mainly to underground combined and separate sewer pipe systems, various methods and technologies are described and discussed. After an introduction to important aspects to deal with when measuring discharges in sewer systems, the following parts are presented successively: (i) measurement of water level with rulers, and pressure, ultrasonic and radar sensors, (ii) measurement of flow velocity with ultrasonic, Doppler, velocity profiler, free surface, and electromagnetic sensors, (iii) direct measurement of discharge with pre-calibrated devices, physical scale models, computational fluid dynamics modelling and use of pumping stations, and (iv) detection and/or measurement of infiltration into and exfiltration from sewers, with flow or pressure measurements, tracer experiments, distributed temperature sensing and geophysical methods.

Designing a monitoring network or a measuring set-up or a monitoring station is a typical (multidisciplinary) engineering enterprise: a range of potentially conflicting demands (technical, financial and managerial) and limitations (e.g. availability of resources, skilled personnel, regulations) have to be respected. This chapter addresses the design aspects on both the macro scale (a monitoring network) and on the micro scale. The macro scale addresses what to measure, where to measure, how frequently to measure and the applications of models in the design process. On the micro scale issues with safety, accessibility and practical limitations are discussed. This chapter has close links with virtually all other chapters in this book and a comprehensive set of literature references is supplied to allow the interested reader to broaden his/her knowledge on the subject.

Vortices are a very common phenomenon to consider in many hydraulic engineering problems, e.g. when designing pump sumps or intake works for turbines. Until now, the focus is on avoiding the development of vortices because of induction of cavitation and air entrainment by a fully developed air core. However, vortices may also be used in a more positive manner e.g. by taking advantage of the capacity to transport (floating) particles to avoid the built up of scum layers in wastewater pump sumps. To the authors’ knowledge only little literature is available in this specific field of research. In order to obtain state-of-the art data on the dynamics of large particles in a free-surface vortex flow field and to examine the vortex transport capacity, a simple but novel 3D-PTV (Particle Tracking Velocimetry) method using 6 iPhones was designed, built, tested and employed to measure the 3D-motion of a large particle in the vortex flow. The experimental set-up, particle detection method and the post-processing of the raw data are described in detail along with some first experimental results. The presented 3D-PTV method showed to be able to obtain data that can be used for detailed analysis of the dynamics of large particles in the vortex flow. The overall standard uncertainty of the particle centre position is in the order of magnitude of 0.1 mm in each direction in a volume of a diameter of 0.6 m and a height of about 1 m, while recording at 240 fps.

Sewer asset management gained momentum and importance in recent years due to economic considerations, since infrastructure maintenance and rehabilitation directly represent major investments. Because physical urban water infrastructure has life expectancies of up to 100 years or more, contemporary urban drainage systems are strongly influenced by historical decisions and implementations. The current decisions taken in sewer asset management will, therefore, have a long-lasting impact on the functionality and quality of future services provided by these networks. These decisions can be supported by different approaches ranging from various inspection techniques, deterioration models to assess the probability of failure or the technical service life, to sophisticated decision support systems crossing boundaries to other urban infrastructure. This paper presents the state of the art in sewer asset management in its manifold facets spanning a wide field of research and highlights existing research gaps while giving an outlook on future developments and research areas.

The deployment of smart metrological procedures and practices as well as challenging operating conditions for urban water monitoring require ever more efficient tools. In this context, the OTHU IField Observatory for Urban Water Management) rehabilitated its field monitoring stations. This paper aims at demonstrating benefits from innovative OTHU's monitoring stations regarding the reliability of collected data and the decrease of maintenance costs. The current technology offers real opportunities in terms of reduced costs, reactivity and improvement of data pre-processing. Each station produces approximately 12,240 data per day, ensuring continuous monitoring of stormwater quantity and quality in urban environment. The long term monitoring of the dynamics of hydrologicat processes requires 24/7 operational and reliable equipment and devices. The new OTHU monitoring stations are reliable and provide representative data of higher quality than the previous ones.

In order to test the laser-scanning device presented by Stanić, Lepot, Catieau, Langeveld and Clemens [1], laboratory experiments have been performed. Various objects, covering a wide range of sizes, shapes, materials, colours and reflectiveness, have been installed in a concrete pipe and scanned by the prototype in order to identify potential object characteristics that may affect measurement uncertainty and/or create bias. By taking into account both uncertainties (on scanned and measured sizes), scanned dimensions have been compared to measurements performed with a calliper or a ruler: overall the values are mutually consistent. The proposed prototype is suitable for sewer inspections: displaced joints, cracks, deposits can be accurately measured without any bias by comparison to CCTV. Uncertainty in the measurement appears to be unaffected by humidity or fat deposits.

A thorough review has been performed on interpolation methods to fill gaps in time-series, efficiency criteria, and uncertainty quantifications. On one hand, there are numerous available methods: interpolation, regression, autoregressive, machine learning methods, etc. On the other hand, there are many methods and criteria to estimate efficiencies of these methods, but uncertainties on the interpolated values are rarely calculated. Furthermore, while they are estimated according to standard methods, the prediction uncertainty is not taken into account: a discussion is thus presented on the uncertainty estimation of interpolated/extrapolated data. Finally, some suggestions for further research and a new method are proposed.

In order to ensure and extend the serviceability of sewage infrastructure, sewer managers need more accurate data to estimate its actual state. For that purpose, various inspection techniques can be implemented. A first prototype of an unbiased laser profiler was developed to improve the accuracy of collected data (Clemens, Stanić, Van der Schoot, Langeveld, Lepot, 2015). However, a higher accuracy needs to be achieved. This article presents the results of an improved prototype that delivers an increase in the measuring accuracy of 5–10 times, which was achieved by using more accurate equipment (lasers and cameras) and by improvements in the methodologies applied (e.g. removing the last bias image distortion by camera calibration). The new design presented hereafter provides accurate measurements (σ < 2 mm) of cross section and accurate 3D image of a pipe. The potential applications of the improved laser profiling technique are comprehensive: e.g. deposit measurements, roughness measurements.

UV/Vis spectrophotometers have been used to monitor water quality since the early 2000s. Calibration of these devices requires sampling campaigns to elaborate relations between recorded spectra and measured concentrations. In order to build robust calibration data sets, several spectra must be recorded per sample. This study compares two approaches – principal component analysis and data depth theory – to identify outliers and select the most representative spectrum (MRS) among the repetitively recorded spectra. Detection of samples that contain outliers is consistent between the methods in more than 70% of the samples. Identification of spectra as outliers is consistent in more than 95% of the cases. The identification of MRS differs depending on the approach used. In their current form, both of the proposed approaches can be used for outlier detection and identification. Further studies are suggested to combine the methods and develop an automated ranking and sorting system.

Separate sewer systems are sensitive to illegal or mis-connections. Several techniques (including the Distributed Temperature Sensor) are now available to identify and locate those connections. Based on thermal fingerprints, DTS allows the localization of each lateral connection along a reach. The use of Infra-Red camera has been investigated with 748 laboratory experiments (artificial connections along a flume). The tested connections vary in diameters (from 75 to 200 mm), lengths of intrusion (from 0 to 200 m), shapes (circular or linear i.e. cracks), depths, discharge rates between the lateral connection and the main flume, and temperatures. IR frame analysis (for detection) and 2D temperature mapping (at the free water surface, for quantification) demonstrate that: i) the detection limit is very low (ratio between lateral and main discharges: 0.025) and ii) the quantification of the lateral discharge is impossible. Application of an IR camera seems to be a promising technique to detect lateral connections.

UV/Vis spectrophotometers have been used for one decade to monitor water quality in various locations: sewers, rivers, wastewater treatment plants (WWTPs), tap water networks, etc. Resulting equivalent concentrations of interest can be estimated by three ways: i) by manufacturer global calibration; ii) by local calibration based on the provided global calibration and grab sampling; iii) by advanced calibration looking for relations between UV/Vis spectra and corresponding concentrations from grab sampling. However, no study has compared the applied methods so far. This collaborative work presents a comparison between five different methods. A Linear Regression (LR), Support Vector Machine (SVM), EVOlutionary algorithm method (EVO) and Partial Least Squares (PLS) have been applied on various data sets (sewers, rivers, WWTPs under dry, wet and all weather conditions) and for three water quality parameters: TSS, COD total and dissolved. Two criteria (r² and Root Mean Square Error RMSE) have been calculated - on calibration and verification data subsets - to evaluate accuracy and robustness of the applied methods. Values of criteria have then been statistically analysed for all and separated data sets. Non-consistent outcomes come through this study. According to the Kruskal-Wallis test and RMSEs, PLS and SVM seem to be the best methods. According to uncertainties in laboratory analysis and ranking of methods, LR and EVO appear more robust and sustainable for concentration estimations. Conclusions are mostly independent of water matrices, weather conditions or concentrations investigated.

Separate sewers are more and more preferred over combined ones but are sensitive to misconnections. Several inspection techniques (time consuming and expensive) are specifically devoted to detecting such misconnections: smoking test and dye tracing are the most popular. This paper presents a new, low-cost and easy to apply method based on 3D printed pills. Numbered (with a unique code) and coloured (blue or red) pills are thrown in toilets, washbowls etc. and then collected in sewers. After an automatic treatment of the collected pills in both the wastewater and the storm water system, the comparison between the sent and collected pill databases allows the identification and the location (inside the building itself) of the misconnections present. Pill and pill analyser designs, analysis methods and on-line tools are described on this paper. Social acceptability and legal issues are briefly discussed.

Monitoring sewer sediments is necessary to better understand sedimentation and erosion processes. Sonar is one of the available techniques to proceed to sewer sediment measurements. Extraction of numerical data, implementation of a new algorithm to identify the water-sediment interface, laboratory and field experiments have been done to evaluate the device, to quantify uncertainties and to test the sonar under various conditions. Results demonstrate that: 1) uncertainties in repeatability are less than 4%, 2) the sonar delivers accurate results under various conditions (small to large sewers and grit chambers), and 3) in situ measurements are affected by uncertainties, mainly due to the fact that the sensor is not in a fixed position but is floating on the free surface in the sewer. This device is useful and accurate for semi-automatic measurement but further research and improvements should be done to better know the position of the device in the section.

In order to propose a new sewer inspection method, a velocity/turbidity profiler (Ubertone, UB Flow) has been tested in a laboratory. A 50 m glass flume has been adapted with several lateral connections (with a arnge of diameter, angle, intrusions, cracks), supplied by a 1 m3 tank. Placed just below the free surface on a rotating (to scan the wet section) and translating (along the main axe of the flume) structure, velocity profiles have been recorded and accurately positioned along the reach (with data from three laser distance meters and a 3 Mpix camera): a 3D cloud of raw velocities is created. After raw data pretreatment (deduction of translation velocity, Nyquist jumps correction, low quality data removal i.e. with low SNR threshold and vector projection corrections), five step-interpolation (adapted from [1]) methods have been implemented and tested: i) data filtering, ii) transformation to flume coordinates velocities, iii) isotropic gridding, iv) anisotropic gridding and v) continuity correction. These methods aim to produce contour lines and the quantification of different streamlines in the reach.

Les réseaux d’assainissement séparatifs sont de plus en plus préférés aux réseaux unitaires mais ils sont sensibles aux erreurs (in)volontaires de branchements entre les bâtiments et le réseau. Il existe quelques techniques d’inspections spécifiques à ce type de recherche : les tests à la fumée et aux
colorants sont les plus utilisés. Ces techniques restent chères et chronophages. Cette étude présente une nouvelle méthode, simple et bon marché, basée sur des pilules imprimées en 3d. Ces pilules, numérotées (avec un code unique) et colorées (bleu ou rouge), sont envoyées par courrier aux habitants, jetées à différents points (toilettes, gouttières, etc.) puis collectées dans les réseaux. Après une analyse automatique des pilules récupérées, une comparaison entre les bases de données des pilules jetées et collectées permet l’identification et la localisation des erreurs de branchements (à l ‘intérieur même des bâtiments). Les conceptions de pilules et de l’analyseur, les méthodes d’analyses et les outils en ligne sont décrits dans ce résumé. L’acceptabilité sociale de la méthode est brièvement discutée. Des tests in situ sont prévus aux Pays-Bas et en France d’ici la conférence.

Separate sewers are more and more preferred than combined ones but are sensitive to (in)voluntary mistakes in the connections between buildings and the network. There are only a few inspection techniques specifically devoted to such investigations: smoking test and dye tracing are the most popular. Those techniques are time consuming and expensive. This paper presents a new, low-cost and easy method based on 3D printed pills. Numbered (with a unique code) and coloured (blue or red) pills are sent by mail to inhabitants, thrown in different locations (toilets, downpipes, etc.) and then collected in sewers. After an automatic treatment of the collected pills, the comparison between the sent and collected pill databases allows the identification and the location (inside the building itself) of the misconnections. Pill and pill analyser designs, analysis methods and on-line tools are described on this paper. Social acceptability is briefly discussed. Some in situ tests will take place in France and in the Netherlands before the conference.

Illicit connections are a major reason of the inefficiency of some separate sewers. Despite theirs costs, dye tracing and smoke test methods are common to investigate mis-connections. An infrared camera (FLIR, A35) was used to identify abnormal thermal finger prints (due to lateral connections) along a flume. The illicit connections were simulated by discharging warm or cold water through several types of lateral connections (variation of diameter, intrusion, depth, etc.). Data analysis revealed that the detection of both illicit connections and groundwater infiltration is possible, albeit under certain conditions.

UV/Vis spectrophotometers have been used to monitor water quality since the early 2000s. Calibration of these devices requires sampling campaigns to elaborate relations between recorded spectra and measured concentrations. Recent sensor improvements allow recordings of a spectrum in as little as 15 seconds, making it possible to record several spectra for the same sample. Spectrum repetitions provide new opportunities to detect outliers – a task that is difficult in non-repetitive spectra recordings. A well-executed outlier detection can e.g. result in a more accurate calibration of the spectrophotometer or an improved construction of a regression model. In this work, two methods are presented and tested to detect outliers in repetitions of spectral data: one based on data depth theory (DDT) and one on principal component analysis (PCA). Results show that the two methods are generally consistent in identifying outliers, with only small differences between the methods.

UV/Vis spectrophotometers have been used to monitor water quality since the early 2000s. Calibration of these devices requires sampling campaigns to elaborate relations between recorded spectra and measured concentrations. Recent sensor improvements allow recordings of a spectrum in as little as 15 seconds, making it possible to record several spectra for the same sample. Spectrum repetitions provide new opportunities to detect outliers – a task that is difficult in non-repetitive spectra recordings. A well-executed outlier detection can e.g. result in a more accurate calibration of the spectrophotometer or an improved construction of a regression model. In this work, two methods are presented and tested to detect outliers in repetitions of spectral data: one based on data depth theory (DDT) and one on principal component analysis (PCA). Results show that the two methods are generally consistent in identifying outliers, with only small differences between the methods.