Uploaded image for project: 'OODT (Retired)'
  1. OODT (Retired)
  2. OODT-215

Workflow2 Architecture

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.5
    • workflow manager
    • None
    • from JPL's internal JIRA, pre-ASF

    Description

      We'll use this issue to keep track of the proposal for a Workflow2 architecture, supporting more complex workflows (fan-ins and fan-outs) and more fine-grained control of the underlying control and data flow models. Here is the proposal from Brian and Paul and me:

      In the new workflow model structure, a model has an execution type. cas-workflow will come standard with 4 workflow execute types implemented, but extendable so that new types can be added, current types can be extended, and even overriding of current types. The standard workflow execution types are (keep in mind that a task is considered a workflow of 1 in the new model):

      1) Sequential

      will run all containing workflows in sequence
      2) Parallel
      will run all containing workflows in parallel
      3) Subset-Parallel
      will run all containing workflows in parallel, but allowing for the case that some containing workflows may not run
      4) Ntimes
      will run all containing workflows in parallel a given number of times determined by a creation rule interface

      Properties can be added to any given workflow thus allowing for a customizable workflow type extension point.

      The default type will be 'Sequential' thus allowing for the old workflow structure to work seamlessly with the new version.

      Example xml:

      <workflow id=""> or <workflow id-ref=""> . . . the former declares that a workflow is being defined; the latter that a workflow has already been defined and is being referenced.

      additional attributes include:
      execution : valid values, by default, are the standard workflow execution types described above
      trans-in : a list of pre-Condition ids that must be met in order for the workflow to run
      trans-out : a set of post-Condition ids that must be met for successful completion of the workflow
      any attribute in the namespace 'p' : this declares a properties for the workflow that can be used to customize this workflow model's execution type
      <task id=""> or <task id-ref=""> . . . same as to the 'workflow' element except no 'execution' attribute is allowed — tasks are a specific type of workflow that is execution 'single' (the implied workflow execution type by the 'task' element).
      also allows for all previously supported child elements allowed in old workflow model

      Comments are of course, welcome.

      Attachments

        1. WorkflowEngine.png
          264 kB
          Chris A. Mattmann

        Issue Links

          1.
          WorkflowInstances should have pre-conditions as well Sub-task Resolved Chris A. Mattmann
          2.
          WorkflowConditions should return a WorkflowConditionOutput that contains failReasons Sub-task Resolved Chris A. Mattmann
          3.
          WorkflowConditions should have a timeout Sub-task Resolved Chris A. Mattmann
          4.
          WorkflowConditions should be identifiable as optional or required Sub-task Resolved Chris A. Mattmann
          5.
          Workflow Manager Dynamic Workflows Sub-task Resolved Chris A. Mattmann
          6.
          Support graph-based workflows using nextTask tags Sub-task Resolved Chris A. Mattmann
          7.
          Sub Workflows Sub-task Resolved Chris A. Mattmann
          8.
          Add ability for sequential and parallel task specifications for Workflows Sub-task Resolved Chris A. Mattmann
          9.
          Add lifecycle model supporting transition, done and waiting states Sub-task Resolved Chris A. Mattmann
          10.
          Workflow Editor GUI Sub-task Resolved Chris A. Mattmann
          11.
          SequentialWorkflowProcessor doesn't need to be a Thread Sub-task Resolved Chris A. Mattmann
          12.
          Port WEngine to trunk Sub-task Resolved Chris A. Mattmann
          13.
          Create a Non-Blocking threaded implementation of the Workflow Engine Sub-task Resolved Chris A. Mattmann
          14.
          NonBlockWorkflow double processing and not pausing Sub-task Resolved Chris A. Mattmann
          15.
          Workflow Priority Sorting Sub-task Resolved Chris A. Mattmann
          16.
          Create Runner framework to allow flexible WorkflowTask execution on different runtimes Sub-task Resolved Chris A. Mattmann
          17.
          Port wengine ControlMetadata to cas-pge's PgeMetadata Sub-task Resolved Brian Foster
          18.
          Add CAS-PGE support for multiple Property Adders Sub-task Resolved Brian Foster
          19.
          Convert CAS-PGE metadata keys and workflow statuses from static Strings to enums Sub-task Resolved Brian Foster
          20.
          Port CAS-PGE's PcsMetadataKeys to PgeTaskMetKeys Sub-task Resolved Brian Foster
          21.
          Create Java Logger Handler for CAS-PGE Sub-task Resolved Brian Foster
          22.
          Improve CAS-PGE logging Sub-task Resolved Brian Foster
          23.
          Make PgeConfigBuilder configuration via PgeTaskMetKeys Sub-task Resolved Brian Foster
          24.
          CAS-PGE should fail when product ingests fail Sub-task Resolved Brian Foster
          25.
          Make WorkflowLifecycleManager understand WorkflowStates Sub-task Resolved Chris A. Mattmann
          26.
          Allow for CAS-PGE PgeMetadata to be dumped to an xml file after initialization Sub-task Resolved Brian Foster
          27.
          Port LoggerOutputStream in CAS-PGE to existing LoggerOutputStream in oodt-commons io package Sub-task Resolved Brian Foster
          28.
          Introduce a CAS-Metadata based renaming interface Sub-task Resolved Brian Foster
          29.
          Add FileSize to know metadata fields set by CAS-Crawler Sub-task Resolved Brian Foster
          30.
          Refactor Graph and ParentChildWorkflow class outside of PackagedWorkflowRepository Sub-task Resolved Chris A. Mattmann
          31.
          Allow input data files to be staged in a local directory Sub-task Resolved Brian Foster
          32.
          Add file staging support to CAS-PGE Sub-task Resolved Brian Foster
          33.
          Add file staging support to CAS-PGE's XmlFilePgeConfigBuilder Sub-task Resolved Brian Foster

          Activity

            People

              chrismattmann Chris A. Mattmann
              chrismattmann Chris A. Mattmann
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: