MB
M.T. Bieber
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
10 records found
1
Over the past years, advanced prognostic models and approaches have been developed. Most existing approaches are tailored to one specific system and cannot adaptively be used on different systems. This can lead to years of research and expertise being put into implementing prognostic models without the capacity to predict system failures, either because of a lack of data or data quality or because failure behavior cannot be captured by data-driven models. In addition, prognostic models are often evaluated using metrics only related to the correctness of predictions, preventing meaningful evaluation of operational performance. This paper makes use of a framework that can automatically choose prognostic settings based on specific system data. It simultaneously optimizes the choice of methodologies using metrics that capture multiple aspects of prediction quality. We apply this framework to both a simulated data set and a real aircraft data set to characterize the impact of metrics on the choice of prognostic methodologies. The results show that the choice of optimization metric greatly impacts the output of the generic prognostic framework and the overall performance. In addition, a definition for data suitability is provided and assessed on the aircraft system data sets.
...
Over the past years, advanced prognostic models and approaches have been developed. Most existing approaches are tailored to one specific system and cannot adaptively be used on different systems. This can lead to years of research and expertise being put into implementing prognostic models without the capacity to predict system failures, either because of a lack of data or data quality or because failure behavior cannot be captured by data-driven models. In addition, prognostic models are often evaluated using metrics only related to the correctness of predictions, preventing meaningful evaluation of operational performance. This paper makes use of a framework that can automatically choose prognostic settings based on specific system data. It simultaneously optimizes the choice of methodologies using metrics that capture multiple aspects of prediction quality. We apply this framework to both a simulated data set and a real aircraft data set to characterize the impact of metrics on the choice of prognostic methodologies. The results show that the choice of optimization metric greatly impacts the output of the generic prognostic framework and the overall performance. In addition, a definition for data suitability is provided and assessed on the aircraft system data sets.
Unmasking overestimation
A re-evaluation of deep anomaly detection in spacecraft telemetry
Journal article
(2024)
-
Lars Herrmann, Marie Bieber, Wim J.C. Verhagen, Fabrice Cosson, Bruno F. Santos
As the volume of telemetry data generated by satellites and other complex systems continues to grow, there is a pressing need for more efficient and accurate anomaly detection methods. Current techniques often rely on human analysis and preset criteria, presenting several challenges including the necessity for expert interpretation and continual updates to match the dynamic mission environment. This paper critically examines the use of deep anomaly detection (DAD) methods in addressing these challenges, evaluating their efficacy on real-world spacecraft telemetry data. It exposes limitations in current DAD research, highlighting the tendency for performance results to be overestimated and suggesting that simpler methods can sometimes outperform more complex DAD algorithms. By comparing established metrics for anomaly detection with newly proposed ones, this paper aims to improve the evaluation of DAD algorithms. It underscores the importance of using less accuracy-inflating metrics and offers a comprehensive comparison of DAD methods on popular benchmark datasets and real-life satellite telemetry data. Among the DAD methods examined, the LSTM algorithm demonstrates considerable promise. However, the paper also reveals the potential limitations of this approach, particularly in complex systems that lack a single, clear predictive failure channel. The paper concludes with a series of recommendations for future research, including the adoption of best practices, the need for high-quality, pre-split datasets, and the investigation of other prediction error methods. Through these insights, this paper contributes to the improved understanding and application of DAD methods, ultimately enhancing the reliability and effectiveness of anomaly detection in real-world scenarios.
...
As the volume of telemetry data generated by satellites and other complex systems continues to grow, there is a pressing need for more efficient and accurate anomaly detection methods. Current techniques often rely on human analysis and preset criteria, presenting several challenges including the necessity for expert interpretation and continual updates to match the dynamic mission environment. This paper critically examines the use of deep anomaly detection (DAD) methods in addressing these challenges, evaluating their efficacy on real-world spacecraft telemetry data. It exposes limitations in current DAD research, highlighting the tendency for performance results to be overestimated and suggesting that simpler methods can sometimes outperform more complex DAD algorithms. By comparing established metrics for anomaly detection with newly proposed ones, this paper aims to improve the evaluation of DAD algorithms. It underscores the importance of using less accuracy-inflating metrics and offers a comprehensive comparison of DAD methods on popular benchmark datasets and real-life satellite telemetry data. Among the DAD methods examined, the LSTM algorithm demonstrates considerable promise. However, the paper also reveals the potential limitations of this approach, particularly in complex systems that lack a single, clear predictive failure channel. The paper concludes with a series of recommendations for future research, including the adoption of best practices, the need for high-quality, pre-split datasets, and the investigation of other prediction error methods. Through these insights, this paper contributes to the improved understanding and application of DAD methods, ultimately enhancing the reliability and effectiveness of anomaly detection in real-world scenarios.
Spacecraft and aircraft systems collect health-related data continuously, which can give an indication of the systems' health status. While they rarely occur, the repercussions of such system anomalies, faults, or failures can be severe, safety-critical and costly. Therefore, the data are used to anticipate any kind of anomalous behaviour or predict remaining useful life. Most of the existing approaches are tailored to one specific system. They do not provide a high degree of flexibility and often cannot be adaptively used on different systems. This can lead to years of research, knowledge and expertise being put in the implementation of prognostics models without the capacity to estimate remaining useful life of systems either because of lack of data or data quality or simply because failure behaviour cannot be captured by data-driven models. To overcome this, in this thesis we present an adaptive diagnostic and prognostic framework which can be applied to different systems while providing a way to assess whether or not it makes sense to put more time into the development of diagnostic or prognostic models for a system. In addition, we apply this framework to both, an aircraft and satellite data set to characterize the impact of metrics on the choice of diagnostic and prognostic methodologies. We highlight considerations for suitability assessment of systems data towards prognostics and diagnostics and show that the choice of optimization metric has a big impact on the output of the generic diagnostic and prognostic framework. The results indicate that such a framework can gives directions for practitioners as to whether or not it makes sense to invest time and money in the development of diagnostic or prognostic systems based on the available system data.
...
Spacecraft and aircraft systems collect health-related data continuously, which can give an indication of the systems' health status. While they rarely occur, the repercussions of such system anomalies, faults, or failures can be severe, safety-critical and costly. Therefore, the data are used to anticipate any kind of anomalous behaviour or predict remaining useful life. Most of the existing approaches are tailored to one specific system. They do not provide a high degree of flexibility and often cannot be adaptively used on different systems. This can lead to years of research, knowledge and expertise being put in the implementation of prognostics models without the capacity to estimate remaining useful life of systems either because of lack of data or data quality or simply because failure behaviour cannot be captured by data-driven models. To overcome this, in this thesis we present an adaptive diagnostic and prognostic framework which can be applied to different systems while providing a way to assess whether or not it makes sense to put more time into the development of diagnostic or prognostic models for a system. In addition, we apply this framework to both, an aircraft and satellite data set to characterize the impact of metrics on the choice of diagnostic and prognostic methodologies. We highlight considerations for suitability assessment of systems data towards prognostics and diagnostics and show that the choice of optimization metric has a big impact on the output of the generic diagnostic and prognostic framework. The results indicate that such a framework can gives directions for practitioners as to whether or not it makes sense to invest time and money in the development of diagnostic or prognostic systems based on the available system data.
Spacecraft systems collect health-related data continuously, which can give an indication of the systems’ health status. While they rarely occur, the repercussions of such system anomalies, faults, or failures can be severe, safety-critical and costly. Therefore, the data are used to anticipate any kind of anomalous behaviour. Typically this is performed by the use of simple thresholds or statistical techniques. Over the past few years, however, data-driven anomaly detection methods have been further developed and improved. They can help to automate the process of anomaly detection. However, it usually is time intensive and requires expertise to identify and implement suitable anomaly detection methods for specific systems, which is often not feasible for application at scale, for instance, when considering a satellite consisting of numerous systems and many more subsystems. To address this limitation, a generic diagnostic framework is proposed that identifies optimal anomaly detection techniques and data pre-processing and thresholding methods. The framework is applied to two publicly available spacecraft datasets and a real-life satellite dataset provided by the European Space Agency. The results show that the framework is robust and adaptive to different system data, providing a quick way to assess anomaly detection for the underlying system. It was found that including thresholding techniques significantly influences the quality of resulting anomaly detection models. With this, the framework can provide both a way forward in developing data-driven anomaly detection methods for spacecraft systems and guidance relative to the direction of anomaly detection method selection and implementation for specific use cases.
...
Spacecraft systems collect health-related data continuously, which can give an indication of the systems’ health status. While they rarely occur, the repercussions of such system anomalies, faults, or failures can be severe, safety-critical and costly. Therefore, the data are used to anticipate any kind of anomalous behaviour. Typically this is performed by the use of simple thresholds or statistical techniques. Over the past few years, however, data-driven anomaly detection methods have been further developed and improved. They can help to automate the process of anomaly detection. However, it usually is time intensive and requires expertise to identify and implement suitable anomaly detection methods for specific systems, which is often not feasible for application at scale, for instance, when considering a satellite consisting of numerous systems and many more subsystems. To address this limitation, a generic diagnostic framework is proposed that identifies optimal anomaly detection techniques and data pre-processing and thresholding methods. The framework is applied to two publicly available spacecraft datasets and a real-life satellite dataset provided by the European Space Agency. The results show that the framework is robust and adaptive to different system data, providing a quick way to assess anomaly detection for the underlying system. It was found that including thresholding techniques significantly influences the quality of resulting anomaly detection models. With this, the framework can provide both a way forward in developing data-driven anomaly detection methods for spacecraft systems and guidance relative to the direction of anomaly detection method selection and implementation for specific use cases.
In recent decades, the increased use of sensor technologies, as well as the increase in digitalisation of aircraft sustainment and operations, have enabled capabilities to detect, diagnose, and predict the health of aircraft structures, systems, and components. Predictive maintenance and closely related concepts, such as prognostics and health management (PHM) have attracted increasing attention from a research perspective, encompassing a growing range of original research papers as well as review papers. When considering the latter, several limitations remain, including a lack of research methodology definition, and a lack of review papers on predictive maintenance which focus on military applications within a defence context. This review paper aims to address these gaps by providing a systematic two-stage review of predictive maintenance focused on a defence domain context, with particular focus on the operations and sustainment of fixed-wing defence aircraft. While defence aircraft share similarities with civil aviation platforms, defence aircraft exhibit significant variation in operations and environment and have different performance objectives and constraints. The review utilises a systematic methodology incorporating bibliometric analysis of the considered domain, as well as text processing and clustering of a set of aligned review papers to position the core topics for subsequent discussion. This discussion highlights state-of-the-art applications and associated success factors in predictive maintenance and decision support, followed by an identification of practical and research challenges. The scope is primarily confined to fixed-wing defence aircraft, including legacy and emerging aircraft platforms. It highlights that challenges in predictive maintenance and PHM for researchers and practitioners alike do not necessarily revolve solely on what can be monitored, but also covers how robust decisions can be made with the quality of data available.
...
In recent decades, the increased use of sensor technologies, as well as the increase in digitalisation of aircraft sustainment and operations, have enabled capabilities to detect, diagnose, and predict the health of aircraft structures, systems, and components. Predictive maintenance and closely related concepts, such as prognostics and health management (PHM) have attracted increasing attention from a research perspective, encompassing a growing range of original research papers as well as review papers. When considering the latter, several limitations remain, including a lack of research methodology definition, and a lack of review papers on predictive maintenance which focus on military applications within a defence context. This review paper aims to address these gaps by providing a systematic two-stage review of predictive maintenance focused on a defence domain context, with particular focus on the operations and sustainment of fixed-wing defence aircraft. While defence aircraft share similarities with civil aviation platforms, defence aircraft exhibit significant variation in operations and environment and have different performance objectives and constraints. The review utilises a systematic methodology incorporating bibliometric analysis of the considered domain, as well as text processing and clustering of a set of aligned review papers to position the core topics for subsequent discussion. This discussion highlights state-of-the-art applications and associated success factors in predictive maintenance and decision support, followed by an identification of practical and research challenges. The scope is primarily confined to fixed-wing defence aircraft, including legacy and emerging aircraft platforms. It highlights that challenges in predictive maintenance and PHM for researchers and practitioners alike do not necessarily revolve solely on what can be monitored, but also covers how robust decisions can be made with the quality of data available.
In recent years, there has been an enormous increase in the amount of research in the field of prognostics and predictive maintenance for mechanical and electrical systems. Most of the existing approaches are tailored to one specific system. They do not provide a high degree of flexibility and often cannot be adaptively used on different systems. This can lead to years of research, knowledge, and expertise being put in the implementation of prognostics models without the capacity to estimate the remaining useful life of systems, either because of lack of data or data quality or simply because failure behaviour cannot be captured by data-driven models. To overcome this, in this paper we present an adaptive prognostic framework which can be applied to different systems while providing a way to assess whether or not it makes sense to put more time into the development of prognostic models for a system. The framework incorporates steps necessary for prognostics, including data pre-processing, feature extraction and machine learning algorithms for remaining useful life estimation. The framework is applied to two systems: a simulated turbofan engine dataset and an aircraft cooling unit dataset. The results show that the obtained accuracy of the remaining useful life estimates are comparable to what has been achieved in literature and highlight considerations for suitability assessment of systems data towards prognostics.
...
In recent years, there has been an enormous increase in the amount of research in the field of prognostics and predictive maintenance for mechanical and electrical systems. Most of the existing approaches are tailored to one specific system. They do not provide a high degree of flexibility and often cannot be adaptively used on different systems. This can lead to years of research, knowledge, and expertise being put in the implementation of prognostics models without the capacity to estimate the remaining useful life of systems, either because of lack of data or data quality or simply because failure behaviour cannot be captured by data-driven models. To overcome this, in this paper we present an adaptive prognostic framework which can be applied to different systems while providing a way to assess whether or not it makes sense to put more time into the development of prognostic models for a system. The framework incorporates steps necessary for prognostics, including data pre-processing, feature extraction and machine learning algorithms for remaining useful life estimation. The framework is applied to two systems: a simulated turbofan engine dataset and an aircraft cooling unit dataset. The results show that the obtained accuracy of the remaining useful life estimates are comparable to what has been achieved in literature and highlight considerations for suitability assessment of systems data towards prognostics.
Metrics play an important part in the development and application of
prognostic methodologies as they provide the capability to characterize
and assess the quality of remaining useful life predictions. Although
there is a wide range of both, prognostic metrics and prognostic
methodologies available, the choice of those often is a demanding and
time consuming task. Additionally, they are often treated as two
separate problems to solve, while the choice of metrics has an impact on
the choice of prognostic methodology and vice versa. In this paper, we
therefore present a framework with the capability to automatically
choose prognostic settings given specific system data to account for
five different prognostic metrics. We then apply this framework to an
aircraft data set to characterize the impact of metrics on the choice of
prognostic methodologies. The results show that the choice of
optimization metric has a big impact on the output of the generic
prognostic framework and on the overall prognostic performance.
...
Metrics play an important part in the development and application of
prognostic methodologies as they provide the capability to characterize
and assess the quality of remaining useful life predictions. Although
there is a wide range of both, prognostic metrics and prognostic
methodologies available, the choice of those often is a demanding and
time consuming task. Additionally, they are often treated as two
separate problems to solve, while the choice of metrics has an impact on
the choice of prognostic methodology and vice versa. In this paper, we
therefore present a framework with the capability to automatically
choose prognostic settings given specific system data to account for
five different prognostic metrics. We then apply this framework to an
aircraft data set to characterize the impact of metrics on the choice of
prognostic methodologies. The results show that the choice of
optimization metric has a big impact on the output of the generic
prognostic framework and on the overall prognostic performance.
Conference paper
(2022)
-
I. Tseremoglou, M.T. Bieber, W.J.C. Verhagen, Bruno F. Santos, F.C. Freeman, P.J. van Kessel
One of the challenges of Condition-Based Maintenance (CBM) is to combine health monitoring and predictions with efficient scheduling tools. However, the majority of literature is focusing on the assessment of prognostics algorithms performance. In fact, the added value of these algorithms can only be assessed when considering their impact on maintenance decision process. Furthermore, in practice, when considering the scenario of an aircraft fleet with multiple monitored components, it is hard for a human decision-maker to translate and identify the effect of probabilistic results from all prognostics models from all systems on the maintenance schedule. Therefore, to support the implementation of CBM, the prognostics algorithms have to be integrated within a scheduling framework. Our paper proposes this integration in order to evaluate the impact of different level of prognostics accuracy and uncertainty on the aircraft fleet maintenance scheduling level. First, a Support Vector Regression (SVR) model is used to predict the Remaining Useful Life (RUL) distributions of the monitored components. Second, the maintenance scheduling problem is solved within a Reinforcement Learning (RL) approach incorporating a state-of-the-art Partially Observable Monte Carlo algorithm. Implementing a rolling horizon approach, our proposed framework is applied to a fleet of 10 aircraft, each equipped with multiple monitored systems. A case study with multiple different prediction accuracy and uncertainty scenarios is performed to assess the impact of prognostics uncertainty on optimal maintenance scheduling. The performed analysis aims to guide the development and assessment of prognostic models in terms of accuracy and uncertainty in the context of CBM.
...
One of the challenges of Condition-Based Maintenance (CBM) is to combine health monitoring and predictions with efficient scheduling tools. However, the majority of literature is focusing on the assessment of prognostics algorithms performance. In fact, the added value of these algorithms can only be assessed when considering their impact on maintenance decision process. Furthermore, in practice, when considering the scenario of an aircraft fleet with multiple monitored components, it is hard for a human decision-maker to translate and identify the effect of probabilistic results from all prognostics models from all systems on the maintenance schedule. Therefore, to support the implementation of CBM, the prognostics algorithms have to be integrated within a scheduling framework. Our paper proposes this integration in order to evaluate the impact of different level of prognostics accuracy and uncertainty on the aircraft fleet maintenance scheduling level. First, a Support Vector Regression (SVR) model is used to predict the Remaining Useful Life (RUL) distributions of the monitored components. Second, the maintenance scheduling problem is solved within a Reinforcement Learning (RL) approach incorporating a state-of-the-art Partially Observable Monte Carlo algorithm. Implementing a rolling horizon approach, our proposed framework is applied to a fleet of 10 aircraft, each equipped with multiple monitored systems. A case study with multiple different prediction accuracy and uncertainty scenarios is performed to assess the impact of prognostics uncertainty on optimal maintenance scheduling. The performed analysis aims to guide the development and assessment of prognostic models in terms of accuracy and uncertainty in the context of CBM.
Prognostics for condition-based maintenance does not only consist of prognostic algorithms but also involves steps such as data pre-processing, feature extraction, and feature selection, all of which contribute to the quality of the remaining useful life estimation. This process requires a lot of expertise and technical knowledge, which for many application systems is neither feasible nor affordable. In this paper, therefore, we present a generic framework with the capability to automatically choose the optimal settings for prognostics, given a specific data set. The framework consists of two phases. In the first one, a genetic algorithm optimizes the choice of methodologies together with hyperparameter settings for the feature extraction, feature selection, and prognostic algorithm selection. In the second phase, the identified settings define the prognostic setup, which in turn is used to train the model for remaining useful life estimation. This framework is then applied to a simulated aircraft engine data set. The first results show that remaining useful life estimates are comparable to the values obtained using established prognostic algorithms on the same data set. In addition, the framework is applied to estimate the remaining useful life of real aircraft systems. Results on underlying data sets suggest that the generic prognostic framework can easily and quickly be adapted to various systems. In further consequence, such a generic framework offers a way to assess the feasibility of prognostics for systems depending on the underlying existing data sets.
...
Prognostics for condition-based maintenance does not only consist of prognostic algorithms but also involves steps such as data pre-processing, feature extraction, and feature selection, all of which contribute to the quality of the remaining useful life estimation. This process requires a lot of expertise and technical knowledge, which for many application systems is neither feasible nor affordable. In this paper, therefore, we present a generic framework with the capability to automatically choose the optimal settings for prognostics, given a specific data set. The framework consists of two phases. In the first one, a genetic algorithm optimizes the choice of methodologies together with hyperparameter settings for the feature extraction, feature selection, and prognostic algorithm selection. In the second phase, the identified settings define the prognostic setup, which in turn is used to train the model for remaining useful life estimation. This framework is then applied to a simulated aircraft engine data set. The first results show that remaining useful life estimates are comparable to the values obtained using established prognostic algorithms on the same data set. In addition, the framework is applied to estimate the remaining useful life of real aircraft systems. Results on underlying data sets suggest that the generic prognostic framework can easily and quickly be adapted to various systems. In further consequence, such a generic framework offers a way to assess the feasibility of prognostics for systems depending on the underlying existing data sets.
During flights aircraft continuously collect data regarding operations, health status and system condition. Data-driven approaches typically applied to system specific sensor data provide a way to predict failures of aircraft systems. However, it is believed that some systems deteriorate faster when subjected to particular environmental conditions, such as humidity or dust. In this study, we consider an aircraft system which is suspected to experience degradation due to humidity during ground operations. We apply a Random Forest approach to sensor data only and a combination of sensor data and environmental data from airports to estimate the system's remaining useful life. To our knowledge this is the first paper addressing the problem of integrating environmental data in prognostics for aircraft systems using raw sensor data. The method is validated on a data set provided by an airline that includes the per-second sensor data of 11 different sensors for roughly 12,300 flights, as well as 15 removals. Meteorological data for airports worldwide is obtained from the Meteorological Aerodrome Reports database. The results show that incorporating environmental data in prognostics has a potential towards more accurate prediction models.
...
During flights aircraft continuously collect data regarding operations, health status and system condition. Data-driven approaches typically applied to system specific sensor data provide a way to predict failures of aircraft systems. However, it is believed that some systems deteriorate faster when subjected to particular environmental conditions, such as humidity or dust. In this study, we consider an aircraft system which is suspected to experience degradation due to humidity during ground operations. We apply a Random Forest approach to sensor data only and a combination of sensor data and environmental data from airports to estimate the system's remaining useful life. To our knowledge this is the first paper addressing the problem of integrating environmental data in prognostics for aircraft systems using raw sensor data. The method is validated on a data set provided by an airline that includes the per-second sensor data of 11 different sensors for roughly 12,300 flights, as well as 15 removals. Meteorological data for airports worldwide is obtained from the Meteorological Aerodrome Reports database. The results show that incorporating environmental data in prognostics has a potential towards more accurate prediction models.