Print Email Facebook Twitter Workflow Automation for Cycling Systems Title Workflow Automation for Cycling Systems Author Oliver, Hilary (The National Institute of Water and Atmospheric Research) Shin, Matthew (Met Office) Matthews, David (Met Office) Sanders, Oliver (Met Office) Bartholomew, Sadie (Met Office) Clark, Andrew (Met Office) Fitzpatrick, Ben (Met Office) Haren, R. van (Netherlands eScience Center) Hut, R.W. (TU Delft Water Resources) Drost, Niels (Netherlands eScience Center) Date 2019 Abstract Complex cycling workflows are fundamental to numerical weather prediction (NWP) and related environmental forecasting systems. Large numbers of jobs are executed at regular intervals to process new data and generate new forecasts. Dependence between these forecast cycles creates a single never-ending workflow, but NWP workflow schedulers have traditionally ignored this-at the cost of efficiency when running “off the clock”-by enforcing a simpler nonoverlapping sequence of single-cycle workflows. Cylc (“Silk”)1 -3 is designed to manage infinite cycling workflows efficiently even after delays in real-time operation, or in historical runs, when cycles can typically interleave for much-increased throughput. Cylc is not actually specialized to environmental forecasting, however, and cycling workflows may also be useful in other contexts. In this paper, we describe the origins and major features of Cylc, future plans for the project, and our experience of Open Source development and community engagement. Subject automationgeophysics computingweather forecastingworkflow management software To reference this document use: http://resolver.tudelft.nl/uuid:4417f7eb-d111-4534-b706-e272af49925d DOI https://doi.org/10.1109/MCSE.2019.2906593 ISSN 1558-366X Source Computing in Science & Engineering, 21 (4), 7-21 Bibliographical note Accepted Author Manuscript Part of collection Institutional Repository Document type journal article Rights © 2019 Hilary Oliver, Matthew Shin, David Matthews, Oliver Sanders, Sadie Bartholomew, Andrew Clark, Ben Fitzpatrick, R. van Haren, R.W. Hut, Niels Drost Files PDF CiSE_Cylc_AuthorAcceptedM ... script.pdf 1.88 MB Close viewer /islandora/object/uuid:4417f7eb-d111-4534-b706-e272af49925d/datastream/OBJ/view