Harder, better, faster, stronger: understanding and improving the tractability of large energy system models

Background: Energy system models based on linear programming have been growing in size with the increasing need to model renewables with high spatial and temporal detail. Larger models lead to high computational requirements. Furthermore, seemingly small changes in a model can lead to drastic differences in runtime. Here, we investigate...

The NLBSE'23 Tool Competition

We report on the organization and results of the second edition of the tool competition from the International Workshop on Natural Language-based Software Engineering (NLBSE'23). As in the prior edition, we organized the competition on automated issue report classification, with a larger dataset. This year, we featured an extra competition on...

Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO

We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. We summarize the competition design...

A numerical benchmark for modelling phase change in molten salt reactors

The design of a molten salt reactor is largely based on CFD simulations. Phase change plays an important role in the safety of the reactor, but numerical modelling of phase change is particularly challenging. Therefore, the knowledge of the margin of error of CFD simulations involving phase change is very important. Relevant experimental...

Gromit: Benchmarking the Performance and Scalability of Blockchain Systems

The growing number of implementations of blockchain systems stands in stark contrast with still limited research on a systematic comparison of performance characteristics of these solutions. Such research is crucial for evaluating fundamental trade-offs introduced by novel consensus protocols and their implementations. These performance...

Analysis of ESAFORM 2021 cup drawing benchmark of an Al alloy, critical factors for accuracy and efficiency of FE simulations

This article details the ESAFORM Benchmark 2021. The deep drawing cup of a 1 mm thick, AA 6016-T4 sheet with a strong cube texture was simulated by 11 teams relying on phenomenological or crystal plasticity approaches, using commercial or self-developed Finite Element (FE) codes, with solid, continuum or classical shell elements and different...

Comparing open data benchmarks: Which metrics and methodologies determine countries’ positions in the ranking lists?

An understanding of the similar and divergent metrics and methodologies underlying open government data benchmarks can reduce the risks of the potential misinterpretation and misuse of benchmarking outcomes by policymakers, politicians, and researchers. Hence, this study aims to compare the metrics and methodologies used to measure, benchmark,...

HistSFC: Optimization for nD massive spatial points querying

Space Filling Curve (SFC) mapping-based clustering and indexing works effectively for point clouds management and querying. It maps both points and queries into a one-dimensional SFC space so that B+- tree could be utilized. Based on the basic structure, this paper develops a generic HistSFC approach which utilizes a histogram tree recording...

Microbially mediated kinetic sulfur isotope fractionation: reactive transport modeling benchmark

Microbially mediated sulfate reduction is a ubiquitous process in many subsurface systems. Isotopic fractionation is characteristic of this anaerobic process, since sulfate-reducing bacteria (SRB) favor the reduction of the lighter sulfate isotopologue (S³²O₄²⁻) over the heavier isotopologue (S³⁴O...

Results from a multi-physics numerical benchmark for codes dedicated to molten salt fast reactors

Verification and validation of multi-physics codes dedicated to fast-spectrum molten salt reactors (MSR) is a very challenging task. Existing benchmarks are meant for single-physics codes, while experimental data for validation are absent. This is concerning, given the importance numerical simulations have in the development of fast MSR...

A comparison of automatic cell identification methods for single-cell RNA sequencing data

Background: Single-cell transcriptomics is rapidly advancing our understanding of the cellular composition of complex tissues and organisms. A major limitation in most analysis pipelines is the reliance on manual annotations to determine cell identities, which are time-consuming and irreproducible. The exponential growth in the number of...

Management of large indoor point clouds: An initial exploration

Indoor navigation and visualization become increasingly important nowadays. Meanwhile, the proliferation of new sensors as well as the advancement of data processing provide massive point clouds to model the indoor environment in high accuracy. However, current state-of-the-art solutions fail to manage such large datasets efficiently. File...

A benchmark and comparison of active learning for logistic regression

Logistic regression is by far the most widely used classifier in real-world applications. In this paper, we benchmark the state-of-the-art active learning methods for logistic regression and discuss and illustrate their underlying characteristics. Experiments are carried out on three synthetic datasets and 44 real-world datasets, providing...

Towards a top-K SPARQL query benchmark generator

The research on optimization of top-k SPARQL query would largely benefit from the establishment of a benchmark that allows comparing different approaches. For such a benchmark to be meaningful, at least two requirements should hold: 1) the benchmark should resemble reality as much as possible, and 2) it should stress the features of the topk...