Residential heating offers many-low hanging fruit for automation and optimization, especially when integrated into a wider Internet of Things (IoT) ecosystem. Typical appliances work in isolation providing minimal controls, such as a thermostat, or extending them into their own domain, for example with daily and weekly settings. Here, I describe how I integrated diverse appliances to obtain intelligence and value beyond that offered by their individual controllers.

Performance is a critical attribute of system software since even small improvements are amplified across the countless CPU instructions devoted to it. In the two previous installments of this column, I described how I ported the Unix sed stream editor1 from C into Rust2 and the system’s design.3 Here, I describe how I optimized its input/output (I/O) performance by exposing advanced operating system (OS) facilities as Rust abstractions.

The C preprocessor, a key element of the language, has become a liability due to its lack of integration with modern language semantics. This column describes the analysis of the C preprocessor usage in the Linux kernel, comprising 20 million lines of code, using the CScout refactoring browser. Processing limitations led to a solution leveraging a supercomputer’s parallel processing capabilities. The analysis divided the kernel’s source files across 32 supercomputer nodes and implemented a binary tournament database merging strategy. Initial efforts revealed multiple difficulties. Resolving them involved several false starts involving recursive SQL statements, an SQLite extension, and the GraphViz connected components tool. After a number of redesigns guided by stress-testing, the analysis finished in just 32 hours rather than a week, using 374 CPU hours and 640 GiB RAM on the supercomputer’s nodes.

RDBUnit is a unit testing framework designed to test relational database queries, created out of a need for unit testing them while working on software analytics tasks. It is available as a Python package on PyPI and open-source software on GitHub. RDBUnit tests consist of three parts: setup, query, and expected result, with the input and output defined as table contents. The framework utilizes a domain-specific language (DSL) for test specifications, employs a simple parsing mechanism, and uses a class hierarchy for managing database differences. It evaluates test results through SQL code generated and handled by the database engine. RDBUnit supports SQLite, mySQL, and PostgreSQL, and is implemented as a command-line tool suitable for diverse operating systems and continuous integration environments. It has proved beneficial in identifying subtle bugs and facilitating a focused and efficient approach to experimenting with SQL queries, especially in big data scenarios, signifying the assurance provided by unit testing in SQL-centric tasks.

Effective data processing workflows are crucial in data science, business analytics, and machine learning. Domain-specific tools can be invaluable, but often custom workflows are needed. Key to their success is splitting data and tasks into manageable chunks to enhance reliability, troubleshooting, and parallelization. Avoid monolithic programs; instead, favor modular designs that simplify data management and processing. Utilizing tools like xargs and GNU parallel can leverage multiple cores or hosts efficiently. Logging and documenting your workflow are essential for monitoring progress and understanding the process. Handling data subsets allows for quicker feedback and testing. Prepare for invalid data and system failures by designing processes that can gracefully manage exceptions and ensure results are reproducible and incremental, avoiding over-engineering. Simplify where possible, leveraging powerful, mature Unix tools and focusing optimization efforts on parts of the code responsible for the bulk of runtime costs. Adhere to software engineering practices to maintain the quality and integrity of your workflow, ensuring it remains a reliable asset to your organization.

Code refactoring is an essential part of software development, because it reduces technical debt, enhances long-term code sustainability, and enables the implementation of functionality that might have been incompatible with an original design. IDEs automate many refactoring tasks, but they sometimes lack support for specific operations or languages. In such cases, regular expressions offer a powerful alternative, automating tedious tasks, reducing errors, and saving time. This article shares a practical example: extending the CScout refactoring browser to collect metrics on C preprocessor usage, which required addressing widespread cyclic dependencies. The changes were facilitated by the "git-subst"Git extension, which makes global text replacements using regular expressions in Git-managed files. A series of 30 git-subst invocations were automatically generated, again using regular expression replacements. While not a cure-all, regular expressions are invaluable for many refactoring tasks, making them a key skill for software developers.

Effective change management is crucial for businesses heavily reliant on software and services to minimise incidents induced by changes. Unfortunately, in practice it is often difficult to effectively use artificial intelligence for IT Operations (AIOps) to enhance service management, primarily due to inadequate data quality. Establishing reliable links between changes and the induced incidents is crucial for identifying patterns, improving change deployment, identifying high-risk changes, and enhancing incident response. In this research, we investigate the enhancement of traceability between changes and incidents through AIOps methods. Our approach involves a close examination of incident-inducing changes, the replication of methods linking incidents to the changes that caused them, introducing an adapted method, and demonstrating its results using historical data and practical evaluations. Our findings reveal that incident-inducing changes exhibit different characteristics dependent on context. Furthermore, a significant disparity exists between assessments based on historical data and real-world observation, with an increased occurrence of false positives when identifying links between unlabeled changes and incidents. This study highlights the complex nature of identifying links between changes and incidents, emphasising the contextual influence on AIOps method effectiveness. While we are actively working on improving the quality of current data through AIOps approaches, it remains apparent that further measures are necessary to address issues like data imbalances and promote a postmortem culture that brings attention to the value of properly administrating tickets. A better overview of change failure rates contributes to improved risk compliance and reliable change management.

Machine learning (ML) techniques increase the effectiveness of software engineering (SE) lifecycle activities. We systematically collected, quality-assessed, summarized, and categorized 83 reviews in ML for SE published between 2009 and 2022, covering 6,117 primary studies. The SE areas most tackled with ML are software quality and testing, while human-centered areas appear more challenging for ML. We propose a number of ML for SE research challenges and actions, including conducting further empirical validation and industrial studies on ML, reconsidering deficient SE methods, documenting and automating data collection and pipeline processes, reexamining how industrial practitioners distribute their proprietary data, and implementing incremental ML approaches.

Developers and data scientists often struggle to write command-line inputs, even though graphical interfaces or tools like ChatGPT can assist. The solution? "ai-cli,"an open-source system inspired by GitHub Copilot that converts natural language prompts into executable commands for various Linux command-line tools. By tapping into OpenAI's API, which allows interaction through JSON HTTP requests, "ai-cli"transforms user queries into actionable command-line instructions. However, integrating AI assistance across multiple command-line tools, especially in open source settings, can be complex. Historically, operating systems could mediate, but individual tool functionality and the lack of a unified approach have made centralized integration challenging. The "ai-cli"tool, by bridging this gap through dynamic loading and linking with each program's Readline library API, makes command-line interfaces smarter and more user-friendly, opening avenues for further enhancement and cross-platform applicability.

We propose a testing framework for validating static typing procedures in compilers. Our core component is a program generator suitably crafted for producing programs that are likely to trigger typing compiler bugs. One of our main contributions is that our program generator gives rise to transformation-based compiler testing for finding typing bugs. We present two novel approaches (type erasure mutation and type overwriting mutation) that apply targeted transformations to an input program to reveal type inference and soundness compiler bugs respectively. Both approaches are guided by an intra-procedural type inference analysis used to capture type information flow. We implement our techniques as a tool, which we call Hephaestus. The extensibility of Hephaestus enables us to test the compilers of three popular JVM languages: Java, Kotlin, and Groovy. Within nine months of testing, we have found 156 bugs (137 confirmed and 85 fixed) with diverse manifestations and root causes in all the examined compilers. Most of the discovered bugs lie in the heart of many critical components related to static typing, such as type inference.