GSD's External Workflow Manager (WFM)

Installation

Download external workflow manager, version 2.0 (svn r43) (1.5 Mb)
Unzip file in a directory of your choice, then do the following (where $WFM_DIR is where you unzipped the WFM files):
cd $WFM_DIR/libxml-ruby-1.1.3/ext/libxml
ruby extconf.rb
make
cd to $WFM_DIR and change the .rb files to executable (chmod +x *.rb)

Introduction (written by Chris Harrop)

Workflow Management is a relatively old concept that originated in the 1970's in the business world to handle business process management. Workflow management systems were developed to manage complex collections of business processes that needed to be carried out in a certain way with complex interdependencies and requirements. Scientific Workflow Management is much newer, and is very much like its business counterpart, except that it is usually data oriented instead of process oriented. That is, scientific workflows are driven by the data that "flows" through them. Workflow tasks are usually triggered by the availability of some kind of input data, and a task's result is usally some kind of data that is eventually fed to another task in the workflow. The individual tasks themselves are scientific codes that perform some kind of computation or retreive or store some type of data for a computation. So, whereas a business workflow is comprised of a diverse set of processes that have to be completed in a certain way, sometimes carried out by a machine, sometimes carried out by a human being, a scientific workflow is usually comprised of a set of computations that are solely driven by the availabiilty of input data.

For more information, visit: http://rdhpcs.noaa.gov/workflowmanager/

Why Workflow Management?

The day when a scientist could conduct his or her numerical modeling and simulation research by writing, running, and monitoriing the progress of a modest Fortran code or two, is quickly becoming a distant memory. It is a fact that researchers now often have to make hundreds or thousands of runs of a numerical model to get a single result. In addition, each "run" of the model often entails running many different codes for pre- and post-processing in addition to the model itself. All the codes that comprise the end-to-end modeling system must be run in a very specific order and there can be complex interdependencies. Furthermore, these codes are usually run on high performance machines that are notoriously difficult for scientists to use, and which tend to suffer from various types of failures. Managing the execution of the end-to-end system is often difficult even for a single run of the end-to-end system. Multiply that by the thousands of runs needed to get the job done, and you have a very challenging situation. For simulations that must run reliably in realtime, the situtation is almost hopeless. A Workflow Management System solves this problem by providing two things:

  1. A means by which to describe the various codes that need to be run, along with their runtime requirements and interdependencies.
  2. A software engine for automating the execution of the workflow

The workflow management system described here uses a custom XML language for defining workflows. The workflows described by those XML files are executed by a workflow management system dubbed "The Workflow Manager", which is really just a set of libraries written in Ruby that interface to the batch system and do all the book keeping necessary to track the progress of the workflow.

The Workflow Manager Language

The Workflow Manager uses a very simple XML language to define workflows. It is specifically targeted for weather and climate simulations, and is not intended to be a catch-all for any type of simulation research. Therefore, it may not be amenable to certain types of research.

XML Header

Every XML file must have this text at the top:

<?xml version="1.0"?>
<!DOCTYPE workflow
[
]>

This is required by XML parsers, and it tells the parsers what kind of document this is. It has no relevance to anything else in the workflow, and can be pretty much ignored.

ENTITIES

The only exception to the above, is that it is often very nice to define constants, called ENTITIES, that can be referenced in other parts of the document. The use of ENTITIES is crucial for creating documents that are easy to maintain. The idea is that you use the ENTITY to represent a value that is used in lots of places in the XML. Then, if you ever have to change it, you only need to change it in one place. The definition of the ENTITIES go between the [ and the ] in the XML header shown above. For example,

<?xml version="1.0"?>
<!DOCTYPE workflow
[
<!ENTITY RUC_HOME "/lfs0/projects/rtruc/13km/backup">
]>

In the above example, an ENTITY called RUC_HOME is defined and given a value of /lfs0/projects/rtruc/13km/backup. Now, anytime we need to use /lfs0/projects/rtruc/13km/backup later in the document, we can use the RUC_HOME ENTITY instead. Then, if the ENTITY definition for RUC_HOME is changed, all places where it is referenced will see the updated value. This has huge benefits for building XML documents that are easy to maintain.

An ENTITY is referenced by the following syntax: &ENTITY_NAME;

And, you can reference an ENTITY when defining another ENTITY. For example,

<?xml version="1.0"?>
<!DOCTYPE workflow
[
<!ENTITY RUC_HOME "/lfs0/projects/rtruc/13km/backup">
<!ENTITY LOG "&RUC_HOME;/log">
]

In the above example, the LOG ENTITY references the RUC_HOME entity in its definition. So, changing the value of RUC_HOME will also change the value of LOG. ENTITIES can be used almost anywhere, but can not be used to hold the values of tag or attribute names. The can be used to hold attribute values, however. For example,

<task id="hybext" action="&KSH_DIR;/hybext.ksh" cycle="6hr,12hr" scheduler="sge" throttle="5" tries="3">   

In the above example, the KSH_DIR ENTITY is used when defining the value of the action attribute in a task tag.

The workflow tag

This is the main tag for defining workflows. Everything except for the header described above must be contained within the workflow tag. It has a few attributes that are useful.

realtime

The realtime attribute defines whether or not the workflow is to be run in realtime, or in retrospective mode. If realtime is set to "T" or "True", the workflow will be run in realtime. If it is set to "F" or "False" it will be run in retrospective mode. The difference between realtime and retrospective mode is in how cycles are processed. In realtime mode, cycles are processed based on the current time of day. That is, the 00Z cycle for Jan 1, 2009 will start as soon as the current time of day is actually 00Z Jan 1, 2009. It will never start sooner or later than that time. In retrospective mode, there is no time dependency that must be adhered to in order to wait for observations, etc. Thus, cycles will run immediately, in chronological order. Each time the workflow manager runs, it will add a new retrospective cycle to process.

The following example illustrates how to set a workflow to run in realtime mode:

<workflow realtime="T">
  Everything else goes in here
</workflow>

reservation_lead_time

The reservation_lead_time attribute is only valid for realtime workflows. It's purpose is to facilitate the use of pseudo-reservations for realtime workflows. In order to make use of a pseudo-reservation, a task must be submitted in advance of when it is supposed to run. The reservation_lead_time specifies the number of seconds in advance of the cycle time, that a task is allowed to be submitted. Use of this attribute requires some additional things to be done in order to ensure that a task does not run early.

<workflow realtime="T" reservation_lead_time="28800">
  Everything else goes here
</workflow>

maxflowrate

The maxflowrate attribute is only valid for retrospective workflows. It's purpose is to facilitate management of computational and storage resources during a retro run by throttling the number of cycles being processed. The value of this attribute is the number of cycles per hour of wallclock time to add to the processing. The value can be a floating point number. If maxflowrate is 2, then 2 new retro cycles will be added to the workflow manager's processing every hour. The number is always converted into a fraction with units of tasks per hour. So, if maxflowrate is set to 0.33333, then one cycle will be added to the processing every 3 hours. If you know approximately how long it takes to compute one cycle, and you determine how many cycles will fit into the compute and disk space resource allocation, the appropriate maxflowrate can be determined.

<workflow realtime="F" maxflowrate="3.0">
  Everything else goes here
</workflow>

The log tag

The log tag defines the name of the workflow manager log(s). It can be anything, but usually it is best to define the name of the log to be dynamically dependent on the cycle being processed. This will put everything for a particular cycle in its own log file.

<workflow realtime="T" reservation_lead_time="28800">    <log>&LOG;/workflow_<cycle_Y/><cycle_m/><cycle_d/><cycle_H/>.log</log>    Everything else goes here   </workflow> 

In the above example, the workflow log is defined to be named /lfs0/projects/rtruc/13km/backup/log/workflow_yyyymmddhh.log where yyyymmddhh represents the cycle time. Note the use of the LOG ENTITY to represent the path of the log file. The cycle_Y, cycle_m, cycle_d, and cycle_H tags are special tags and are described below.

The cycle tags

It is often necessary to refer to the various components of the current cycle time when defining various aspects of the workflow. For example, when defining the name of the workflow log as in the previous example above, we need to specify the year, month, day, and hour of the cycle. Since the cycle is a dynamic quantity, we can't just put the value in there, we need a special tag to represent it. The cycle tags accomplish this. There is one cycle tag for each component of the cycle time. They are:

<cycle_Y/>   <cycle_y/>   <cycle_m/>   <cycle_d/>   <cycle_j/>   <cycle_H/>  <cycle_M/>  <cycle_S/>

These tags represent the 4-digit year, 2-digit year, month, day, julian day, hour, minute, and second of the cycle time currently being processed. The workflow manager may be processing multiple cycles at once. The workflow manager replaces those tags with the appropriate value as each cycle is processed. These tags can be used in any combination to represent any time desired.

offset

Sometimes, it is necessary to be able to represent not just the current cycle, but some offset before or after the current cycle. This can be accomplished through the use of the offset attribute. The offset attribute is set to the number of seconds from the cycle time desired. For example,

<cycle_Y offset="3600"/><cycle_m offset="3600"/><cycle_d offset="3600"/><cycle_H offset="3600"/>  

The above represents the yyyymmddHH of the current cycle, plus one hour. The offset can be negative. To represent one hour earlier than the cycle time, simply use an offset of -3600.

The cycle tag

The cycle tag defines the set of cycles the workflow is to be run on. The format of a cycle tag is similar to that of a crontab file, and consists of fields for year, month, day, hour, minute, and second. Each field can be a single value, a range of values, a comma separated list of values, etc.

<workflow realtime="T">      <log>&LOG;/workflow_<cycle_Y/><cycle_m/><cycle_d/><cycle_H/>.log</log>      <cycle  id="1hr">2006-2010 * * 1,2,4,5,7,8,10,11,13,14,16,17,19,20,22,23 0 0</cycle>    <cycle  id="3hr">2006-2010 * * 3,9,15,21                                 0 0</cycle>    <cycle  id="6hr">2006-2010 * * 6,18                                      0 0</cycle>    <cycle id="12hr">2006-2010 * * 0,12                                      0 0</cycle>     </workflow>   

The above example shows four cycle tags. Each of these defines a list of years, months, days, hours, minutes, and seconds. A * is shorthand for all values of that field. So, the * in the second field means all months. And the * in the third field means all days. In the example above, the first cycle tag defines a set of cycles consisting of all hours not divisible by three, for every day of every month of years 2006 thru 2010. The second cycle tag defines a set of cycles consisting of hours 3,9,15,21 for every day of every month of years 2006 thru 2010. You can have as many cycle tags as you want. In some cases, it may be necessary to use multiple cycle tags to represent a non-continguous set of cycles. The workflow manager uses the union of all sets of cycles specified by the cycle tags to create the overall cycle pool. If there is overlap between definitions, it will not cause cycles to be run twice.

id

Sometimes it is necessary to define distinct sets of cycles because some tasks should only be run for certain subsets of cycles. The id attribute of the cycle tag allows you to assign a name to a set of cycles that can later be used to control which tasks are run for which cycles. No two cycles may have the same id.

The task tag

The task tag is the bread and butter of the workflow. It defines the computations that you want to run. This is the most complicated tag, and it has several attributes. The example below shows the basic form of the task tag. The contents of the task tag have been left out for now for clarity.
<workflow realtime="T">      <log>&LOG;/workflow_<cycle_Y/><cycle_m/><cycle_d/><cycle_H/>.log</log>      <cycle  id="1hr">2006-2010 * * 1,2,4,5,7,8,10,11,13,14,16,17,19,20,22,23 0 0</cycle>    <cycle  id="3hr">2006-2010 * * 3,9,15,21                                 0 0</cycle>    <cycle  id="6hr">2006-2010 * * 6,18                                      0 0</cycle>    <cycle id="12hr">2006-2010 * * 0,12                                      0 0</cycle>      <task id="hybext" action="&KSH_DIR;/hybext.ksh" cycle="6hr,12hr" scheduler="sge" throttle="5" tries="3">      </task>    </workflow>   

In the example above, a task named hybext has been defined. The command that does this task's work is the hybext.ksh script in the path represented by the &KSH_DIR; ENTITY. The task is set to run for the set of cycles represented by the cycles whose ids are 6hr, and 12hr. The batch scheduler for this task is SGE. No more than 5 instances of this task can be submitted and/or running simultaneously. The workflow manager will make no more than 3 attempts at running this task if it fails. More details are described below.

id

Every task must have a unique id defined. It can be almost anything, but there can never be two tasks in the same workflow with the same id. This id is used to define task dependencies which are described later.

action

Every task must have an action defined. This is simply the name of the script that carries out the task's work. It is the script that the workflow manager will submit to the batch system for execution.

cycle

The cycle attribute is optional. If set, its value must be a comma separated list of cycle tag ids. If it is not set, the task will be run for every cycle in the general pool of cycles. The general pool of cycles is the union of all sets of cycles, both with, and without ids, specified in the cycle tags. If the cycle attribute is set, the task will only be run for cycles that are defined by the cycle tags having the ids listed.

scheduler

The scheduler attribute must be set to one of: sge, lsf, or ll. This attribute tells the workflow manager which batch system to use when managing the task. It is currently not possible to use more than one batch system within the same workflow. On Jet, the value must be sge. At NCAR, the value should be lsf. On a system using LoadLeveler? , the value should be ll (two ells).

throttle

The throttle attribute is optional. If set, it limits the number of instances of that particular task which may be submitted and/or running at the same time. This can be useful for managing disk space or for improving performance where too many copies of a certain task might overwhelm resources. If it is not set, it is treated as if it were set to infinity.

tries

The tries attribute is optional. When the workflow manager detects that a task has failed, it will attempt to resubmit it. The tries attribute limits the number of times a task can be retried before the workflow manager gives up.

HINT: If the number of tries has been exhausted, but you want to retry it one more time, just modify the XML on-the-fly and increment this value. The workflow manager will rerun the task again unless it has expired.

The property tag

The property tag is used inside task tags to define batch system options. The property tag contains a name, value pair that defines the name of the option and its value. Not all properties must have a value, but they must all have a name.

  <task id="hybext" action="&KSH_DIR;/hybext.ksh" cycle="6hr,12hr" scheduler="sge" throttle="5" tries="3">      <property>        <name>-A</name>        <value>&PROJECT;</value>      </property>      <property>        <name>-pe</name>        <value>&PE; 1</value>      </property>      <property>        <name>-V</name>      </property>    </task    

Note the heavy use of ENTITIES in the example above. This is one of the places where ENTITIES are really valuable. The above example illustrates how you can set the -A, -pe, and -V qsub options.

The environment tag

The environment tag is very similar to the property tag. It is used inside task tags to define environment variables that must be passed to the task when it is executed. Like the property tag, it consists of name, value pairs, and the value may be absent.

  <task id="hybext" action="&KSH_DIR;/hybext.ksh" cycle="6hr,12hr" scheduler="sge" throttle="5" tries="3">        <property>        <name>-A</name>        <value>&PROJECT;</value>      </property>      <property>        <name>-pe</name>        <value>&PE; 1</value>      </property>      <property>        <name>-V</name>      </property>      <property>        <name>-cwd</name>      </property>      <property>        <name>-N</name>        <value>hybext_<cycle_H/></value>      </property>      <property>        <name>-o</name>        <value>&LOG;/hybext_<cycle_Y/><cycle_m/><cycle_d/><cycle_H/>.log</value>      </property>      <property>        <name>-j</name>        <value>y</value>      </property>        <environment>        <name>MAPTIME</name>        <value><cycle_y/><cycle_j/><cycle_H/>00</value>      </environment>      <environment>        <name>RUC_HOME</name>        <value>&RUC_HOME;</value>      </environment>  </task>   

In the above example, the MAPTIME and RUC_HOME environment variables will be set to the values given (note the use of the cycle tags and ENTITIES) and passed to the task when it executes. This is the primary means by which the workflow manager passes parameters to the scripts when tasks are run. Command line arguments to scripts are not supported at this time.

The dependency tag

Dependency tags are fundamental to every workflow. They are used inside task tags, and are used to describe the inter-dependencies of the tasks. It is this tag that is used to define things like "task B can't run until task A is complete", or "Task A can't run until this file shows up". Dependencies are defined as boolean expressions, which means that you have quite a lot of control over when tasks are eligible to run. There are three types of dependencies:

  1. Task Dependencies
  2. File Dependencies
  3. Time Dependencies

These dependencies can be combined in any combination in a boolean expression to form a task's overall dependency.

  <task id="hybcst_pre" action="&KSH_DIR;/hybcst_pre.ksh" scheduler="sge" throttle="5" tries="3">        Property and environment tags left out for clarity           <dependency>        <taskdep task="hybpost_analysis"/>      </dependency>      </task>   

The above example shows the basic format of the dependency tag. Inside the dependency tag there must be exactly one tag. The one tag inside the dependency tag must be one of the following tag:

  • taskdep
  • filedep
  • timedep
  • and
  • or
  • not

taskdep

Task dependencies are defined with the taskdep tag as in the following example:

      <taskdep task="hybpost_analysis"/>

The task attribute of the taskdep tag must be set to the id of the task which must complete in order for the dependency to be satisfied.

filedep

File dependencies are defined with the filedep tag as in the following example:

      <filedep age="120">&RUC_HOME;/run/NNT_hybcst/<cycle_y/><cycle_j/><cycle_H/>01.13NNT_dat</filedep>

The age attribute of the filedep tag is optional. It contains the number of seconds the file must not be modified before the file is considered to be available. This is useful for preventing partially written files from erroneously triggering the submission of a task. The value between the start and end of the filedep tag is simply the name of the file. File dependencies are satisfied when the file in question exists and has not been modified for at least age seconds.

timedep

Time dependencies are defined with the timedep tag. The value between the start and end of the timedep tag is a time in yyyymmddhhmmss format. Time dependencies are satisfied with the wall clock time is equal to or greater than the time specified. All times are calculated in GMT.

<timedep><cycle_Y offset="&DEADLINE;"/><cycle_m offset="&DEADLINE;"/><cycle_d offset="&DEADLINE;"/><cycle_H offset="&DEADLINE;"/><cycle_M offset="&DEADLINE;"/>00</timedep>  

and, or, not

The and, or, and not tags are used to compose boolean expressions of dependencies. All dependencies in an and tag must be satisfied for the value of the and tag to be true. At least one dependency in an or tag must be satisfied in order for the value of the or tag to be true. These tags can be nested and combined without limit.

<dependency>
<and>
<timedep><cycle_Y offset="1500"/><cycle_m offset="1500"/><cycle_d offset="1500"/><cycle_H offset="1500"/><cycle_M offset="1500"/><cycle_S offset="1500"/></timedep>
<or>
<timedep><cycle_Y offset="3000"/><cycle_m offset="3000"/><cycle_d offset="3000"/><cycle_H offset="3000"/><cycle_M offset="3000"/><cycle_S offset="3000"/></timedep>

      <filedep age="120">&RUC_HOME;/run/maps_fcst/<cycle_y offset="-3600"/><cycle_j offset="-3600"/><cycle_H offset="-3600"/>000001.grib</filedep>
<and>
<timedep><cycle_Y offset="2700"/><cycle_m offset="2700"/><cycle_d offset="2700"/><cycle_H offset="2700"/><cycle_M offset="2700"/><cycle_S offset="2700"/></timedep>
<or>
<filedep age="120">&RUC_HOME;/run/maps_fcst/<cycle_y offset="-7200"/><cycle_j offset="-7200"/><cycle_H offset="-7200"/>000002.grib</filedep>
<filedep age="120">&RUC_HOME;/run/maps_fcst/<cycle_y offset="-10800"/><cycle_j offset="-10800"/><cycle_H offset="-10800"/>000003.grib</filedep>
<filedep age="120">&RUC_HOME;/run/maps_fcst/<cycle_y offset="-14400"/><cycle_j offset="-14400"/><cycle_H offset="-14400"/>000004.grib</filedep>
<filedep age="120">&RUC_HOME;/run/maps_fcst/<cycle_y offset="-18000"/><cycle_j offset="-18000"/><cycle_H offset="-18000"/>000005.grib</filedep>
<filedep age="120">&RUC_HOME;/run/maps_fcst/<cycle_y offset="-21600"/><cycle_j offset="-21600"/><cycle_H offset="-21600"/>000006.grib</filedep>
</or>
</and>
</or>
</and>
</dependency>

deadlinedependency

The deadline dependency tag is used to specify a condition which will permanently terminate a particular task. It uses the same dependency and boolean expression tags as for a normal dependency. The only difference is that if a deadlinedependency is satisfied, the task will abort, any job associated with it will be killed, and no retries will ever be attempted.

hangdependency

The hang dependency tag is used to specify a condition which indicates a task has hung. This is useful for situations where a system or programming error has caused a job to hang. This tag tells the workflow manager how to distinguish between a job that is running normally, and one that is hung. When a hangdependency is satisfied, the job associated with the task is killed. The task will be retried if the maximum retry count has not been exceeded. A common way to use this is to use a file dependency with a relatively long age attribute on an output file with the idea of detecting when a job stops writing output.