Rethinking log parsing in the context of modern software ecosystems

Modern systems generate a tremendous amount of data, making manual investigations infeasible, hence requiring automating the process of analysis. However, running automated log analysis pipelines is far from straightforward, due to the changing nature of software ecosystems caused by the constant need to adapt to user requirements. In practice,...

Inferring Log-Based Behavioural System Models using Markov Chains

An essential step of software development is obtaining an understanding of the behaviour of a system. Accurate state models of system behaviour might help software developers build such an understanding. There exist several techniques for automatically inferring models on system behaviour using log analysis, but these do not scale well for...

Log-based software monitoring: A systematic mapping study

Modern software development and operations rely on monitoring to understand how systems behave in production. The data provided by application logs and runtime environment are essential to detect and diagnose undesired behavior and improve system reliability. However, despite the rich ecosystem around industry- ready log solutions, monitoring...

LogChunks: A Data Set for Build Log Analysis

Build logs are textual by-products that a software build process creates, often as part of its Continuous Integration (CI) pipeline. Build logs are a paramount source of information for developers when debugging into and understanding a build failure. Recently, attempts to partly automate this time-consuming, purely manual activity have come up,...

Log Differencing using State Machines for Anomaly Detection

Huge amounts of log data are generated every day by software. These data contain valuable information about the behavior and the health of the system, which is rarely exploited, because of their volume and unstructured nature. Manually going through log files is a time-consuming and labor-intensive procedure for developers. Nonetheless logging...

Real-time anomaly detection in logs using rule mining and complex event processing at scale

Log data, produced from every computer system and program, are widely used as source of valuable information to monitor and understand their behavior and their health. However, as large-scale systems generate a massive amount of log data every minute, it is impossible to detect the cause of system failure by examining manually this huge size of...

BachelorEnd Project: Real-time anomaly detection in critical Rabobank Processes

Real-time processing of log data can give valuable insights in the behaviour of systems and processes. The Rabobank is a large bank and has several of these systems and processes, one of such is the QR device registration process. In order to monitor this process, an application was built which detects abnormal behaviour. The application reads...

A large-scale evaluation of tracing back log data to its origin with static analysis

Logs are widely used as source of information to understand the activity of computer systems and to monitor their health and stability. As large-scale systems generate hundreds of millions of logs per hour reaching tens of terabytes, automated techniques exist to take advantage of the rich information present in logs. However, these techniques...

Monitoring Release Logs at Adyen: Feature Extraction and Anomaly Detection

Monitoring the release logs of modern online software is a challenging topic because of the enormous amount of release logs and the complicated release process. The goal of this thesis is to develop a pipeline that can monitor the release logs and find anomalous logs, automating this step with anomaly detection and reducing the required manual...

A Search-based Approach for Accurate Identification of Log Message Formats

Many software engineering activities process the events contained in log files. However, before performing any processing activity, it is necessary to parse the entries in a log file, to retrieve the actual events recorded in the log. Each event is denoted by a log message, which is composed of a fixed part—called (event) template—that is the...

Log Analysis from A to Z: A Literature Survey

Using logs to detect and diagnose problems in software systems is no longer a feasible human process. The ever increasing amount of logs produced by present-day systems calls for more advanced techniques to enable log analysis. A great deal of log research has since then been focused on abstracting over log messages, clever analysis techniques...