Workflow Automation for Cycling Systems

Journal article (2019)

Authors

Hilary Oliver The National Institute of Water and Atmospheric Research
Matthew Shin Met Office
D. T A Matthews Met Office
Oliver Sanders Met Office
Sadie Bartholomew Met Office
A Clark Met Office
Ben Fitzpatrick Met Office
R. van Haren Netherlands eScience Center
R.W. Hut Water Resources - CEG
N Drost Netherlands eScience Center
Research Group
Water Resources (CEG) (TU Delft)
Copyright: © 2019 Hilary Oliver, Matthew Shin, David Matthews, Oliver Sanders, Sadie Bartholomew, Andrew Clark, Ben Fitzpatrick, R. van Haren, R.W. Hut, Niels Drost
DOI
help_outline
https://doi.org/10.1109/MCSE.2019.2906593
Automation Geophysics computing Weather forecasting Workflow management software
More Info
expand_more

Published Date

2019

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Civil Engineering & Geosciences

Department

Water Management

Research Group

Water Resources

Abstract

Complex cycling workflows are fundamental to numerical weather prediction (NWP) and related environmental forecasting systems. Large numbers of jobs are executed at regular intervals to process new data and generate new forecasts. Dependence between these forecast cycles creates a single never-ending workflow, but NWP workflow schedulers have traditionally ignored this-at the cost of efficiency when running “off the clock”-by enforcing a simpler nonoverlapping sequence of single-cycle workflows. Cylc (“Silk”)1 -3 is designed to manage infinite cycling workflows efficiently even after delays in real-time operation, or in historical runs, when cycles can typically interleave for much-increased throughput. Cylc is not actually specialized to environmental forecasting, however, and cycling workflows may also be useful in other contexts. In this paper, we describe the origins and major features of Cylc, future plans for the project, and our experience of Open Source development and community engagement.