D. Spinellis
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
158 records found
1
...
Source Code Hotspots
A Diagnostic Method for Quality Issues
Software source code often harbours "hotspots"- small portions of the code that change far more often than the rest of the project and thus concentrate maintenance activity. We mine the complete version histories of 91 evolving, actively developed GitHub repositories and identify 15 recurring line-level hotspot patterns that explain why these hotspots emerge. The three most prevalent patterns are Pinned Version Bump (26%), revealing brittle release practices; Long Line Change (17%), signalling deficient layout; and Formatting Ping-Pong (9%), indicating missing or inconsistent style automation. Surprisingly, automated accounts generate 74% of all hotspot edits, suggesting that bot activity is a dominant - but largely avoidable - source of noise in change histories. By mapping each pattern to concrete refactoring guidelines and continuous integration checks, our taxonomy equips practitioners with actionable steps to curb hotspots and systematically improve software quality in terms of configurability, stability, and changeability.
...
Software source code often harbours "hotspots"- small portions of the code that change far more often than the rest of the project and thus concentrate maintenance activity. We mine the complete version histories of 91 evolving, actively developed GitHub repositories and identify 15 recurring line-level hotspot patterns that explain why these hotspots emerge. The three most prevalent patterns are Pinned Version Bump (26%), revealing brittle release practices; Long Line Change (17%), signalling deficient layout; and Formatting Ping-Pong (9%), indicating missing or inconsistent style automation. Surprisingly, automated accounts generate 74% of all hotspot edits, suggesting that bot activity is a dominant - but largely avoidable - source of noise in change histories. By mapping each pattern to concrete refactoring guidelines and continuous integration checks, our taxonomy equips practitioners with actionable steps to curb hotspots and systematically improve software quality in terms of configurability, stability, and changeability.
Digital identity systems face an inherent tension between verifiability, non-transferability, and privacy. While current European Digital Identity Wallet prototypes rely on Selective-Disclosure JWTs (SD-JWTs) to achieve minimal disclosure and backward compatibility, these designs fail to uphold the strict three-way unlinkability requirement mandated by EU regulations. Schemes based on BBS signatures have been proposed as a possible solution to this problem. While these schemes satisfy the stated requirements, their use of complex cryptographic operations incurs a non-negligible performance overhead, especially relevant for mobile devices. To empirically investigate this overhead, we construct an extendable benchmarking suite for end-to-end wallet operations. Using our suite, we perform experiments comparing the performance of several privacy-preserving verifiable credential implementations on a desktop computer, a smartphone, a smartwatch, and a low-cost single-board computer. We find that performance varies significantly (often by an order of magnitude) based on hardware, employed scheme, and programming language of the implementation. Performance differences between schemes and implementations are more pronounced on resource-constrained devices. Positively, with the best-performing unlinkable BBS variant, a smartwatch needs around 1.5 seconds for credential presentation, latency that can be considered practically acceptable. Our results inform discussions on the transition to privacy-preserving mobile identity wallets, while our open-source suite can be used to benchmark future implementations.
...
Digital identity systems face an inherent tension between verifiability, non-transferability, and privacy. While current European Digital Identity Wallet prototypes rely on Selective-Disclosure JWTs (SD-JWTs) to achieve minimal disclosure and backward compatibility, these designs fail to uphold the strict three-way unlinkability requirement mandated by EU regulations. Schemes based on BBS signatures have been proposed as a possible solution to this problem. While these schemes satisfy the stated requirements, their use of complex cryptographic operations incurs a non-negligible performance overhead, especially relevant for mobile devices. To empirically investigate this overhead, we construct an extendable benchmarking suite for end-to-end wallet operations. Using our suite, we perform experiments comparing the performance of several privacy-preserving verifiable credential implementations on a desktop computer, a smartphone, a smartwatch, and a low-cost single-board computer. We find that performance varies significantly (often by an order of magnitude) based on hardware, employed scheme, and programming language of the implementation. Performance differences between schemes and implementations are more pronounced on resource-constrained devices. Positively, with the best-performing unlinkable BBS variant, a smartwatch needs around 1.5 seconds for credential presentation, latency that can be considered practically acceptable. Our results inform discussions on the transition to privacy-preserving mobile identity wallets, while our open-source suite can be used to benchmark future implementations.
A Git extension identifies source-code hotspots by analyzing code churn and line lifetimes across repository histories. Its development illustrates extending Git, transforming a research prototype into a Python package, integrating shell pipelines, configurable output formatting, and lessons from agentic AI software development.
...
A Git extension identifies source-code hotspots by analyzing code churn and line lifetimes across repository histories. Its development illustrates extending Git, transforming a research prototype into a Python package, integrating shell pipelines, configurable output formatting, and lessons from agentic AI software development.
Performance is a critical attribute of system software since even small improvements are amplified across the countless CPU instructions devoted to it. In the two previous installments of this column, I described how I ported the Unix sed stream editor1 from C into Rust2 and the system’s design.3 Here, I describe how I optimized its input/output (I/O) performance by exposing advanced operating system (OS) facilities as Rust abstractions.
...
Performance is a critical attribute of system software since even small improvements are amplified across the countless CPU instructions devoted to it. In the two previous installments of this column, I described how I ported the Unix sed stream editor1 from C into Rust2 and the system’s design.3 Here, I describe how I optimized its input/output (I/O) performance by exposing advanced operating system (OS) facilities as Rust abstractions.
Residential heating offers many-low hanging fruit for automation and optimization, especially when integrated into a wider Internet of Things (IoT) ecosystem. Typical appliances work in isolation providing minimal controls, such as a thermostat, or extending them into their own domain, for example with daily and weekly settings. Here, I describe how I integrated diverse appliances to obtain intelligence and value beyond that offered by their individual controllers.
...
Residential heating offers many-low hanging fruit for automation and optimization, especially when integrated into a wider Internet of Things (IoT) ecosystem. Typical appliances work in isolation providing minimal controls, such as a thermostat, or extending them into their own domain, for example with daily and weekly settings. Here, I describe how I integrated diverse appliances to obtain intelligence and value beyond that offered by their individual controllers.
Generative artificial intelligence (AI) coding tools are transforming software production and work. To benefit, organizations must integrate AI into processes, enforce stricter quality controls, manage automation risks, and adapt training, career paths, and governance to sustained AI-assisted development.
...
Generative artificial intelligence (AI) coding tools are transforming software production and work. To benefit, organizations must integrate AI into processes, enforce stricter quality controls, manage automation risks, and adapt training, career paths, and governance to sustained AI-assisted development.
Conference paper
(2026)
-
Ioannis Karyotakis, Foivos Timotheos Proestakis, Evangelos Talos, Diomidis Spinellis, Nikolaos Alexopoulos
Mobile messaging apps are central to user privacy, yet their practical implementations remain understudied. Using a hybrid static-dynamic methodology, we compare the Android clients of Meta Messenger, Signal, and Telegram. Our results show clear differences: Signal has the smallest attack surface, Messenger exhibits extensive background activity, and Telegram requests the most high-risk permissions. All three apps comply with the Android permission model.
...
Mobile messaging apps are central to user privacy, yet their practical implementations remain understudied. Using a hybrid static-dynamic methodology, we compare the Android clients of Meta Messenger, Signal, and Telegram. Our results show clear differences: Signal has the smallest attack surface, Messenger exhibits extensive background activity, and Telegram requests the most high-risk permissions. All three apps comply with the Android permission model.
Learning from Change
Predictive Models for Incident Prevention in a Regulated IT Environment
Conference paper
(2026)
-
Eileen Kapel, Jan Lennartz, Luis Cruz, Diomidis Spinellis, Arie Van Deursen
Effective IT change management is important for businesses that depend on software and services, particularly in highly regulated sectors such as finance, where operational reliability, auditability, and explainability are essential. A significant portion of IT incidents are caused by changes, making it important to identify high-risk changes before deployment. This study presents a predictive incident risk scoring approach at a large international bank. The approach supports engineers during the assessment and planning phases of change deployments by predicting the potential of inducing incidents. To satisfy regulatory constraints, we built the model with auditability and explainability in mind, applying SHAP values to provide feature-level insights and ensure decisions are traceable and transparent. Using a one-year real-world dataset, we compare the existing rule-based process with three machine learning models: HGBC, LightGBM, and XGBoost. LightGBM achieved the best performance, particularly when enriched with aggregated team metrics that capture organisational context. Our results show that data-driven, interpretable models can outperform rule-based approaches while meeting compliance needs, enabling proactive risk mitigation and more reliable IT operations.
...
Effective IT change management is important for businesses that depend on software and services, particularly in highly regulated sectors such as finance, where operational reliability, auditability, and explainability are essential. A significant portion of IT incidents are caused by changes, making it important to identify high-risk changes before deployment. This study presents a predictive incident risk scoring approach at a large international bank. The approach supports engineers during the assessment and planning phases of change deployments by predicting the potential of inducing incidents. To satisfy regulatory constraints, we built the model with auditability and explainability in mind, applying SHAP values to provide feature-level insights and ensure decisions are traceable and transparent. Using a one-year real-world dataset, we compare the existing rule-based process with three machine learning models: HGBC, LightGBM, and XGBoost. LightGBM achieved the best performance, particularly when enriched with aggregated team metrics that capture organisational context. Our results show that data-driven, interpretable models can outperform rule-based approaches while meeting compliance needs, enabling proactive risk mitigation and more reliable IT operations.
IN CONTRAST TO physical objects and living things, software doesn’t deteriorate with the passage of time. While we age and our shoes fall apart, digital storage ensures that the software’s bits stay immutable. And yet, software needs substantial maintenance over time, owing to changes in its environment.1 Advancing technology and new requirements prompt us to modernize the software to keep it relevant. Here, I show how these changes happen in practice by describing the evolution and modernization of a burglar alarm security system I first developed a quarter-century ago. […]
...
IN CONTRAST TO physical objects and living things, software doesn’t deteriorate with the passage of time. While we age and our shoes fall apart, digital storage ensures that the software’s bits stay immutable. And yet, software needs substantial maintenance over time, owing to changes in its environment.1 Advancing technology and new requirements prompt us to modernize the software to keep it relevant. Here, I show how these changes happen in practice by describing the evolution and modernization of a burglar alarm security system I first developed a quarter-century ago. […]
The Unix sed stream editor is a programmable text processing filter, first written in C in the 1970s.1 In the previous installment of this column, I described the tool and how I reimplemented it in Rust with some help from generative AI.2 Here, I describe the new implementation’s design to show how we can create a simple programmable tool. In the next “Adventures in Code” column, I’ll present optimizations that substantially increased the tool’s throughput.
What’s behind a programmable tool, such as Python, sed, or SQLite? It turns out that the key component is the data structures used to represent the code. As a student, I came across the equation “Algorithms + Data Structures = Programs.”3 I can still remember that at the time I knew what algorithms were, but data structures were to me a fuzzy concept of academic only interest. “Why would anyone need something more than the numeric and string arrays supported by the BASIC language?” I thought. Over the years, I’ve come to appreciate the value and significance of how we organize our data. Now, I believe that data structures are far more important than algorithms. First, in modern systems, bespoke sophisticated algorithms play only a minor, if any, role; data structures rule! Second, in many cases the algorithm’s choice is a clear-cut decision, whereas deciding on the data structure to use requires a deep understanding of context and a careful weighing of tradeoffs. Third, type systems have matured to offer us powerful practical aid in dealing with data; corresponding formal models for algorithms less so.
Consequently, the design of many systems should often start by considering how data will be structured: a database’s schema, key classes and their fields, types and operations on them. ...
What’s behind a programmable tool, such as Python, sed, or SQLite? It turns out that the key component is the data structures used to represent the code. As a student, I came across the equation “Algorithms + Data Structures = Programs.”3 I can still remember that at the time I knew what algorithms were, but data structures were to me a fuzzy concept of academic only interest. “Why would anyone need something more than the numeric and string arrays supported by the BASIC language?” I thought. Over the years, I’ve come to appreciate the value and significance of how we organize our data. Now, I believe that data structures are far more important than algorithms. First, in modern systems, bespoke sophisticated algorithms play only a minor, if any, role; data structures rule! Second, in many cases the algorithm’s choice is a clear-cut decision, whereas deciding on the data structure to use requires a deep understanding of context and a careful weighing of tradeoffs. Third, type systems have matured to offer us powerful practical aid in dealing with data; corresponding formal models for algorithms less so.
Consequently, the design of many systems should often start by considering how data will be structured: a database’s schema, key classes and their fields, types and operations on them. ...
The Unix sed stream editor is a programmable text processing filter, first written in C in the 1970s.1 In the previous installment of this column, I described the tool and how I reimplemented it in Rust with some help from generative AI.2 Here, I describe the new implementation’s design to show how we can create a simple programmable tool. In the next “Adventures in Code” column, I’ll present optimizations that substantially increased the tool’s throughput.
What’s behind a programmable tool, such as Python, sed, or SQLite? It turns out that the key component is the data structures used to represent the code. As a student, I came across the equation “Algorithms + Data Structures = Programs.”3 I can still remember that at the time I knew what algorithms were, but data structures were to me a fuzzy concept of academic only interest. “Why would anyone need something more than the numeric and string arrays supported by the BASIC language?” I thought. Over the years, I’ve come to appreciate the value and significance of how we organize our data. Now, I believe that data structures are far more important than algorithms. First, in modern systems, bespoke sophisticated algorithms play only a minor, if any, role; data structures rule! Second, in many cases the algorithm’s choice is a clear-cut decision, whereas deciding on the data structure to use requires a deep understanding of context and a careful weighing of tradeoffs. Third, type systems have matured to offer us powerful practical aid in dealing with data; corresponding formal models for algorithms less so.
Consequently, the design of many systems should often start by considering how data will be structured: a database’s schema, key classes and their fields, types and operations on them.
What’s behind a programmable tool, such as Python, sed, or SQLite? It turns out that the key component is the data structures used to represent the code. As a student, I came across the equation “Algorithms + Data Structures = Programs.”3 I can still remember that at the time I knew what algorithms were, but data structures were to me a fuzzy concept of academic only interest. “Why would anyone need something more than the numeric and string arrays supported by the BASIC language?” I thought. Over the years, I’ve come to appreciate the value and significance of how we organize our data. Now, I believe that data structures are far more important than algorithms. First, in modern systems, bespoke sophisticated algorithms play only a minor, if any, role; data structures rule! Second, in many cases the algorithm’s choice is a clear-cut decision, whereas deciding on the data structure to use requires a deep understanding of context and a careful weighing of tradeoffs. Third, type systems have matured to offer us powerful practical aid in dealing with data; corresponding formal models for algorithms less so.
Consequently, the design of many systems should often start by considering how data will be structured: a database’s schema, key classes and their fields, types and operations on them.
The C preprocessor, a key element of the language, has become a liability due to its lack of integration with modern language semantics. This column describes the analysis of the C preprocessor usage in the Linux kernel, comprising 20 million lines of code, using the CScout refactoring browser. Processing limitations led to a solution leveraging a supercomputer’s parallel processing capabilities. The analysis divided the kernel’s source files across 32 supercomputer nodes and implemented a binary tournament database merging strategy. Initial efforts revealed multiple difficulties. Resolving them involved several false starts involving recursive SQL statements, an SQLite extension, and the GraphViz connected components tool. After a number of redesigns guided by stress-testing, the analysis finished in just 32 hours rather than a week, using 374 CPU hours and 640 GiB RAM on the supercomputer’s nodes.
...
The C preprocessor, a key element of the language, has become a liability due to its lack of integration with modern language semantics. This column describes the analysis of the C preprocessor usage in the Linux kernel, comprising 20 million lines of code, using the CScout refactoring browser. Processing limitations led to a solution leveraging a supercomputer’s parallel processing capabilities. The analysis divided the kernel’s source files across 32 supercomputer nodes and implemented a binary tournament database merging strategy. Initial efforts revealed multiple difficulties. Resolving them involved several false starts involving recursive SQL statements, an SQLite extension, and the GraphViz connected components tool. After a number of redesigns guided by stress-testing, the analysis finished in just 32 hours rather than a week, using 374 CPU hours and 640 GiB RAM on the supercomputer’s nodes.
Science typically advances in small incremental steps, but in some rare instances it leaps forward. One discovery or invention can change how we see the world around us. Would it not be neat to be able to accurately pinpoint those moments of time in an objective way and thereby investigate science and technology’s progress? In 2016, Russel Funk of the University of Minnesota’s Carlson School of Management and Jason Owen-Smith from the University of Michigan published a measure for exactly this purpose.1 Their so-called consolidation-disruption (CD) index quantifies the extent to which published findings affect the subsequent use of the knowledge on which those findings relied. Worryingly, a widely cited subsequent study applied this measure on patents and scientific publications, finding a slowdown in disruptive progress.2 Thickening the plot, a later preprint attributed the finding to dataset artefacts.3 These studies prompt the need for an efficient way to calculate the CD index on large amounts of openly available data. [...]
...
Science typically advances in small incremental steps, but in some rare instances it leaps forward. One discovery or invention can change how we see the world around us. Would it not be neat to be able to accurately pinpoint those moments of time in an objective way and thereby investigate science and technology’s progress? In 2016, Russel Funk of the University of Minnesota’s Carlson School of Management and Jason Owen-Smith from the University of Michigan published a measure for exactly this purpose.1 Their so-called consolidation-disruption (CD) index quantifies the extent to which published findings affect the subsequent use of the knowledge on which those findings relied. Worryingly, a widely cited subsequent study applied this measure on patents and scientific publications, finding a slowdown in disruptive progress.2 Thickening the plot, a later preprint attributed the finding to dataset artefacts.3 These studies prompt the need for an efficient way to calculate the CD index on large amounts of openly available data. [...]
Background
The proliferation of generative artificial intelligence (AI) has facilitated the creation and publication of fraudulent scientific articles, often in predatory journals. This study investigates the extent of AI-generated content in the Global International Journal of Innovative Research (GIJIR), where a fabricated article was falsely attributed to me.
Methods
The entire GIJIR website was crawled to collect article PDFs and metadata. Automated scripts were used to extract the number of probable in-text citations, DOIs, affiliations, and contact emails. A heuristic based on the number of in-text citations was employed to identify the probability of AI-generated content. A subset of articles was manually reviewed for AI indicators such as formulaic writing and missing empirical data. Turnitin’s AI detection tool was used as an additional indicator. The extracted data were compiled into a structured dataset, which was analyzed to examine human-authored and AI-generated articles.
Results
Of the 53 examined articles with the fewest in-text citations, at least 48 appeared to be AI-generated, while five showed signs of human involvement. Turnitin’s AI detection scores confirmed high probabilities of AI-generated content in most cases, with scores reaching 100% for multiple papers. The analysis also revealed fraudulent authorship attribution, with AI-generated articles falsely assigned to researchers from prestigious institutions. The journal appears to use AI-generated content both to inflate its standing through misattributed papers and to attract authors aiming to inflate their publication record.
Conclusions
The findings highlight the risks posed by AI-generated and misattributed research articles, which threaten the credibility of academic publishing. Ways to mitigate these issues include strengthening identity verification mechanisms for DOIs and ORCIDs, enhancing AI detection methods, and reforming research assessment practices. Without effective countermeasures, the unchecked growth of AI-generated content in scientific literature could severely undermine trust in scholarly communication. ...
The proliferation of generative artificial intelligence (AI) has facilitated the creation and publication of fraudulent scientific articles, often in predatory journals. This study investigates the extent of AI-generated content in the Global International Journal of Innovative Research (GIJIR), where a fabricated article was falsely attributed to me.
Methods
The entire GIJIR website was crawled to collect article PDFs and metadata. Automated scripts were used to extract the number of probable in-text citations, DOIs, affiliations, and contact emails. A heuristic based on the number of in-text citations was employed to identify the probability of AI-generated content. A subset of articles was manually reviewed for AI indicators such as formulaic writing and missing empirical data. Turnitin’s AI detection tool was used as an additional indicator. The extracted data were compiled into a structured dataset, which was analyzed to examine human-authored and AI-generated articles.
Results
Of the 53 examined articles with the fewest in-text citations, at least 48 appeared to be AI-generated, while five showed signs of human involvement. Turnitin’s AI detection scores confirmed high probabilities of AI-generated content in most cases, with scores reaching 100% for multiple papers. The analysis also revealed fraudulent authorship attribution, with AI-generated articles falsely assigned to researchers from prestigious institutions. The journal appears to use AI-generated content both to inflate its standing through misattributed papers and to attract authors aiming to inflate their publication record.
Conclusions
The findings highlight the risks posed by AI-generated and misattributed research articles, which threaten the credibility of academic publishing. Ways to mitigate these issues include strengthening identity verification mechanisms for DOIs and ORCIDs, enhancing AI detection methods, and reforming research assessment practices. Without effective countermeasures, the unchecked growth of AI-generated content in scientific literature could severely undermine trust in scholarly communication. ...
Background
The proliferation of generative artificial intelligence (AI) has facilitated the creation and publication of fraudulent scientific articles, often in predatory journals. This study investigates the extent of AI-generated content in the Global International Journal of Innovative Research (GIJIR), where a fabricated article was falsely attributed to me.
Methods
The entire GIJIR website was crawled to collect article PDFs and metadata. Automated scripts were used to extract the number of probable in-text citations, DOIs, affiliations, and contact emails. A heuristic based on the number of in-text citations was employed to identify the probability of AI-generated content. A subset of articles was manually reviewed for AI indicators such as formulaic writing and missing empirical data. Turnitin’s AI detection tool was used as an additional indicator. The extracted data were compiled into a structured dataset, which was analyzed to examine human-authored and AI-generated articles.
Results
Of the 53 examined articles with the fewest in-text citations, at least 48 appeared to be AI-generated, while five showed signs of human involvement. Turnitin’s AI detection scores confirmed high probabilities of AI-generated content in most cases, with scores reaching 100% for multiple papers. The analysis also revealed fraudulent authorship attribution, with AI-generated articles falsely assigned to researchers from prestigious institutions. The journal appears to use AI-generated content both to inflate its standing through misattributed papers and to attract authors aiming to inflate their publication record.
Conclusions
The findings highlight the risks posed by AI-generated and misattributed research articles, which threaten the credibility of academic publishing. Ways to mitigate these issues include strengthening identity verification mechanisms for DOIs and ORCIDs, enhancing AI detection methods, and reforming research assessment practices. Without effective countermeasures, the unchecked growth of AI-generated content in scientific literature could severely undermine trust in scholarly communication.
The proliferation of generative artificial intelligence (AI) has facilitated the creation and publication of fraudulent scientific articles, often in predatory journals. This study investigates the extent of AI-generated content in the Global International Journal of Innovative Research (GIJIR), where a fabricated article was falsely attributed to me.
Methods
The entire GIJIR website was crawled to collect article PDFs and metadata. Automated scripts were used to extract the number of probable in-text citations, DOIs, affiliations, and contact emails. A heuristic based on the number of in-text citations was employed to identify the probability of AI-generated content. A subset of articles was manually reviewed for AI indicators such as formulaic writing and missing empirical data. Turnitin’s AI detection tool was used as an additional indicator. The extracted data were compiled into a structured dataset, which was analyzed to examine human-authored and AI-generated articles.
Results
Of the 53 examined articles with the fewest in-text citations, at least 48 appeared to be AI-generated, while five showed signs of human involvement. Turnitin’s AI detection scores confirmed high probabilities of AI-generated content in most cases, with scores reaching 100% for multiple papers. The analysis also revealed fraudulent authorship attribution, with AI-generated articles falsely assigned to researchers from prestigious institutions. The journal appears to use AI-generated content both to inflate its standing through misattributed papers and to attract authors aiming to inflate their publication record.
Conclusions
The findings highlight the risks posed by AI-generated and misattributed research articles, which threaten the credibility of academic publishing. Ways to mitigate these issues include strengthening identity verification mechanisms for DOIs and ORCIDs, enhancing AI detection methods, and reforming research assessment practices. Without effective countermeasures, the unchecked growth of AI-generated content in scientific literature could severely undermine trust in scholarly communication.
Effective data processing workflows are crucial in data science, business analytics, and machine learning. Domain-specific tools can be invaluable, but often custom workflows are needed. Key to their success is splitting data and tasks into manageable chunks to enhance reliability, troubleshooting, and parallelization. Avoid monolithic programs; instead, favor modular designs that simplify data management and processing. Utilizing tools like xargs and GNU parallel can leverage multiple cores or hosts efficiently. Logging and documenting your workflow are essential for monitoring progress and understanding the process. Handling data subsets allows for quicker feedback and testing. Prepare for invalid data and system failures by designing processes that can gracefully manage exceptions and ensure results are reproducible and incremental, avoiding over-engineering. Simplify where possible, leveraging powerful, mature Unix tools and focusing optimization efforts on parts of the code responsible for the bulk of runtime costs. Adhere to software engineering practices to maintain the quality and integrity of your workflow, ensuring it remains a reliable asset to your organization.
...
Effective data processing workflows are crucial in data science, business analytics, and machine learning. Domain-specific tools can be invaluable, but often custom workflows are needed. Key to their success is splitting data and tasks into manageable chunks to enhance reliability, troubleshooting, and parallelization. Avoid monolithic programs; instead, favor modular designs that simplify data management and processing. Utilizing tools like xargs and GNU parallel can leverage multiple cores or hosts efficiently. Logging and documenting your workflow are essential for monitoring progress and understanding the process. Handling data subsets allows for quicker feedback and testing. Prepare for invalid data and system failures by designing processes that can gracefully manage exceptions and ensure results are reproducible and incremental, avoiding over-engineering. Simplify where possible, leveraging powerful, mature Unix tools and focusing optimization efforts on parts of the code responsible for the bulk of runtime costs. Adhere to software engineering practices to maintain the quality and integrity of your workflow, ensuring it remains a reliable asset to your organization.
Effective change management is crucial for businesses heavily reliant on software and services to minimise incidents induced by changes. Unfortunately, in practice it is often difficult to effectively use artificial intelligence for IT Operations (AIOps) to enhance service management, primarily due to inadequate data quality. Establishing reliable links between changes and the induced incidents is crucial for identifying patterns, improving change deployment, identifying high-risk changes, and enhancing incident response. In this research, we investigate the enhancement of traceability between changes and incidents through AIOps methods. Our approach involves a close examination of incident-inducing changes, the replication of methods linking incidents to the changes that caused them, introducing an adapted method, and demonstrating its results using historical data and practical evaluations. Our findings reveal that incident-inducing changes exhibit different characteristics dependent on context. Furthermore, a significant disparity exists between assessments based on historical data and real-world observation, with an increased occurrence of false positives when identifying links between unlabeled changes and incidents. This study highlights the complex nature of identifying links between changes and incidents, emphasising the contextual influence on AIOps method effectiveness. While we are actively working on improving the quality of current data through AIOps approaches, it remains apparent that further measures are necessary to address issues like data imbalances and promote a postmortem culture that brings attention to the value of properly administrating tickets. A better overview of change failure rates contributes to improved risk compliance and reliable change management.
...
Effective change management is crucial for businesses heavily reliant on software and services to minimise incidents induced by changes. Unfortunately, in practice it is often difficult to effectively use artificial intelligence for IT Operations (AIOps) to enhance service management, primarily due to inadequate data quality. Establishing reliable links between changes and the induced incidents is crucial for identifying patterns, improving change deployment, identifying high-risk changes, and enhancing incident response. In this research, we investigate the enhancement of traceability between changes and incidents through AIOps methods. Our approach involves a close examination of incident-inducing changes, the replication of methods linking incidents to the changes that caused them, introducing an adapted method, and demonstrating its results using historical data and practical evaluations. Our findings reveal that incident-inducing changes exhibit different characteristics dependent on context. Furthermore, a significant disparity exists between assessments based on historical data and real-world observation, with an increased occurrence of false positives when identifying links between unlabeled changes and incidents. This study highlights the complex nature of identifying links between changes and incidents, emphasising the contextual influence on AIOps method effectiveness. While we are actively working on improving the quality of current data through AIOps approaches, it remains apparent that further measures are necessary to address issues like data imbalances and promote a postmortem culture that brings attention to the value of properly administrating tickets. A better overview of change failure rates contributes to improved risk compliance and reliable change management.
RDBUnit is a unit testing framework designed to test relational database queries, created out of a need for unit testing them while working on software analytics tasks. It is available as a Python package on PyPI and open-source software on GitHub. RDBUnit tests consist of three parts: setup, query, and expected result, with the input and output defined as table contents. The framework utilizes a domain-specific language (DSL) for test specifications, employs a simple parsing mechanism, and uses a class hierarchy for managing database differences. It evaluates test results through SQL code generated and handled by the database engine. RDBUnit supports SQLite, mySQL, and PostgreSQL, and is implemented as a command-line tool suitable for diverse operating systems and continuous integration environments. It has proved beneficial in identifying subtle bugs and facilitating a focused and efficient approach to experimenting with SQL queries, especially in big data scenarios, signifying the assurance provided by unit testing in SQL-centric tasks.
...
RDBUnit is a unit testing framework designed to test relational database queries, created out of a need for unit testing them while working on software analytics tasks. It is available as a Python package on PyPI and open-source software on GitHub. RDBUnit tests consist of three parts: setup, query, and expected result, with the input and output defined as table contents. The framework utilizes a domain-specific language (DSL) for test specifications, employs a simple parsing mechanism, and uses a class hierarchy for managing database differences. It evaluates test results through SQL code generated and handled by the database engine. RDBUnit supports SQLite, mySQL, and PostgreSQL, and is implemented as a command-line tool suitable for diverse operating systems and continuous integration environments. It has proved beneficial in identifying subtle bugs and facilitating a focused and efficient approach to experimenting with SQL queries, especially in big data scenarios, signifying the assurance provided by unit testing in SQL-centric tasks.
Code refactoring is an essential part of software development, because it reduces technical debt, enhances long-term code sustainability, and enables the implementation of functionality that might have been incompatible with an original design. IDEs automate many refactoring tasks, but they sometimes lack support for specific operations or languages. In such cases, regular expressions offer a powerful alternative, automating tedious tasks, reducing errors, and saving time. This article shares a practical example: extending the CScout refactoring browser to collect metrics on C preprocessor usage, which required addressing widespread cyclic dependencies. The changes were facilitated by the "git-subst"Git extension, which makes global text replacements using regular expressions in Git-managed files. A series of 30 git-subst invocations were automatically generated, again using regular expression replacements. While not a cure-all, regular expressions are invaluable for many refactoring tasks, making them a key skill for software developers.
...
Code refactoring is an essential part of software development, because it reduces technical debt, enhances long-term code sustainability, and enables the implementation of functionality that might have been incompatible with an original design. IDEs automate many refactoring tasks, but they sometimes lack support for specific operations or languages. In such cases, regular expressions offer a powerful alternative, automating tedious tasks, reducing errors, and saving time. This article shares a practical example: extending the CScout refactoring browser to collect metrics on C preprocessor usage, which required addressing widespread cyclic dependencies. The changes were facilitated by the "git-subst"Git extension, which makes global text replacements using regular expressions in Git-managed files. A series of 30 git-subst invocations were automatically generated, again using regular expression replacements. While not a cure-all, regular expressions are invaluable for many refactoring tasks, making them a key skill for software developers.
Enhancing Incident Management
Insights from a Case Study at ING
An incident management process is necessary in businesses that depend strongly on software and services. A proper process is essential to guarantee that incidents are well-handled, especially in a financial software-defined business needing to adhere to guidelines and regulations. This paper aims to enhance understanding of the current state of practice through a single-case exploratory case study, at the international bank ING, by interviewing 15 subject matter experts on the incident management process. The research identifies eight core observations on tool usage, the challenges experienced and future opportunities. Core challenges include monitoring data quality, the complexity of the environment, and the balance between minimising incident resolution time and following procedural guidelines. Future opportunities can lessen these challenges by making better use of available tooling and employing machine learning approaches. This requires tight supervision on the use of best practices and good monitoring data quality. The findings emphasise the need for a strengthened focus on improving the quality of monitoring data, handling environment complexity, incident clustering, and better support for regulatory compliance.
...
An incident management process is necessary in businesses that depend strongly on software and services. A proper process is essential to guarantee that incidents are well-handled, especially in a financial software-defined business needing to adhere to guidelines and regulations. This paper aims to enhance understanding of the current state of practice through a single-case exploratory case study, at the international bank ING, by interviewing 15 subject matter experts on the incident management process. The research identifies eight core observations on tool usage, the challenges experienced and future opportunities. Core challenges include monitoring data quality, the complexity of the environment, and the balance between minimising incident resolution time and following procedural guidelines. Future opportunities can lessen these challenges by making better use of available tooling and employing machine learning approaches. This requires tight supervision on the use of best practices and good monitoring data quality. The findings emphasise the need for a strengthened focus on improving the quality of monitoring data, handling environment complexity, incident clustering, and better support for regulatory compliance.
Generative AI based on large-language models is significantly impacting software development through IDE assistants, cloud-based APIs, and interactive chatbots for coding assistance. It excels in generating and translating code and data, navigating APIs, and creating boilerplate content, thereby enhancing productivity. However, it is prone to generating inaccurate information (“hallucinations”), erroneous code, and potentially introducing security vulnerabilities. To counter these risks, employing automated analysis tools, conducting rigorous testing, and maintaining a deep understanding of computer science concepts are essential. While generative AI can substantially aid development tasks it is not a replacement for human expertise, especially in understanding complex software, its requirements, and architecture.
...
Generative AI based on large-language models is significantly impacting software development through IDE assistants, cloud-based APIs, and interactive chatbots for coding assistance. It excels in generating and translating code and data, navigating APIs, and creating boilerplate content, thereby enhancing productivity. However, it is prone to generating inaccurate information (“hallucinations”), erroneous code, and potentially introducing security vulnerabilities. To counter these risks, employing automated analysis tools, conducting rigorous testing, and maintaining a deep understanding of computer science concepts are essential. While generative AI can substantially aid development tasks it is not a replacement for human expertise, especially in understanding complex software, its requirements, and architecture.