Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5
    • Component/s: workflow manager
    • Labels:
      None
    • Environment:

      from JPL's internal JIRA, pre-ASF

      Description

      We'll use this issue to keep track of the proposal for a Workflow2 architecture, supporting more complex workflows (fan-ins and fan-outs) and more fine-grained control of the underlying control and data flow models. Here is the proposal from Brian and Paul and me:

      In the new workflow model structure, a model has an execution type. cas-workflow will come standard with 4 workflow execute types implemented, but extendable so that new types can be added, current types can be extended, and even overriding of current types. The standard workflow execution types are (keep in mind that a task is considered a workflow of 1 in the new model):

      1) Sequential

      will run all containing workflows in sequence
      2) Parallel
      will run all containing workflows in parallel
      3) Subset-Parallel
      will run all containing workflows in parallel, but allowing for the case that some containing workflows may not run
      4) Ntimes
      will run all containing workflows in parallel a given number of times determined by a creation rule interface

      Properties can be added to any given workflow thus allowing for a customizable workflow type extension point.

      The default type will be 'Sequential' thus allowing for the old workflow structure to work seamlessly with the new version.

      Example xml:

      <workflow id=""> or <workflow id-ref=""> . . . the former declares that a workflow is being defined; the latter that a workflow has already been defined and is being referenced.

      additional attributes include:
      execution : valid values, by default, are the standard workflow execution types described above
      trans-in : a list of pre-Condition ids that must be met in order for the workflow to run
      trans-out : a set of post-Condition ids that must be met for successful completion of the workflow
      any attribute in the namespace 'p' : this declares a properties for the workflow that can be used to customize this workflow model's execution type
      <task id=""> or <task id-ref=""> . . . same as to the 'workflow' element except no 'execution' attribute is allowed — tasks are a specific type of workflow that is execution 'single' (the implied workflow execution type by the 'task' element).
      also allows for all previously supported child elements allowed in old workflow model

      Comments are of course, welcome.

      1. WorkflowEngine.png
        264 kB
        Chris A. Mattmann

        Issue Links

        1.
        WorkflowInstances should have pre-conditions as well Sub-task Resolved Chris A. Mattmann
         
        2.
        WorkflowConditions should return a WorkflowConditionOutput that contains failReasons Sub-task Resolved Chris A. Mattmann
         
        3.
        WorkflowConditions should have a timeout Sub-task Resolved Chris A. Mattmann
         
        4.
        WorkflowConditions should be identifiable as optional or required Sub-task Resolved Chris A. Mattmann
         
        5.
        Workflow Manager Dynamic Workflows Sub-task Resolved Chris A. Mattmann
         
        6.
        Support graph-based workflows using nextTask tags Sub-task Resolved Chris A. Mattmann
         
        7.
        Sub Workflows Sub-task Resolved Chris A. Mattmann
         
        8.
        Add ability for sequential and parallel task specifications for Workflows Sub-task Resolved Chris A. Mattmann
         
        9.
        Add lifecycle model supporting transition, done and waiting states Sub-task Resolved Chris A. Mattmann
         
        10.
        Workflow Editor GUI Sub-task Resolved Chris A. Mattmann
         
        11.
        SequentialWorkflowProcessor doesn't need to be a Thread Sub-task Resolved Chris A. Mattmann
         
        12.
        Port WEngine to trunk Sub-task Resolved Chris A. Mattmann
         
        13.
        Create a Non-Blocking threaded implementation of the Workflow Engine Sub-task Resolved Chris A. Mattmann
         
        14.
        NonBlockWorkflow double processing and not pausing Sub-task Resolved Chris A. Mattmann
         
        15.
        Workflow Priority Sorting Sub-task Resolved Chris A. Mattmann
         
        16.
        Create Runner framework to allow flexible WorkflowTask execution on different runtimes Sub-task Resolved Chris A. Mattmann
         
        17.
        Port wengine ControlMetadata to cas-pge's PgeMetadata Sub-task Resolved Brian Foster
         
        18.
        Add CAS-PGE support for multiple Property Adders Sub-task Resolved Brian Foster
         
        19.
        Convert CAS-PGE metadata keys and workflow statuses from static Strings to enums Sub-task Resolved Brian Foster
         
        20.
        Port CAS-PGE's PcsMetadataKeys to PgeTaskMetKeys Sub-task Resolved Brian Foster
         
        21.
        Create Java Logger Handler for CAS-PGE Sub-task Resolved Brian Foster
         
        22.
        Improve CAS-PGE logging Sub-task Resolved Brian Foster
         
        23.
        Make PgeConfigBuilder configuration via PgeTaskMetKeys Sub-task Resolved Brian Foster
         
        24.
        CAS-PGE should fail when product ingests fail Sub-task Resolved Brian Foster
         
        25.
        Make WorkflowLifecycleManager understand WorkflowStates Sub-task Resolved Chris A. Mattmann
         
        26.
        Allow for CAS-PGE PgeMetadata to be dumped to an xml file after initialization Sub-task Resolved Brian Foster
         
        27.
        Port LoggerOutputStream in CAS-PGE to existing LoggerOutputStream in oodt-commons io package Sub-task Resolved Brian Foster
         
        28.
        Introduce a CAS-Metadata based renaming interface Sub-task Resolved Brian Foster
         
        29.
        Add FileSize to know metadata fields set by CAS-Crawler Sub-task Resolved Brian Foster
         
        30.
        Refactor Graph and ParentChildWorkflow class outside of PackagedWorkflowRepository Sub-task Resolved Chris A. Mattmann
         
        31.
        Allow input data files to be staged in a local directory Sub-task Resolved Brian Foster
         
        32.
        Add file staging support to CAS-PGE Sub-task Resolved Brian Foster
         
        33.
        Add file staging support to CAS-PGE's XmlFilePgeConfigBuilder Sub-task Resolved Brian Foster
         

          Activity

          Hide
          Chris A. Mattmann added a comment -
          • all done, we are pretty much all there, thanks guys!
          Show
          Chris A. Mattmann added a comment - all done, we are pretty much all there, thanks guys!
          Hide
          Chris A. Mattmann added a comment -

          haha you like that? Maybe I was using FILO prioritizer hahah. As for the queues, yeah one of them might be queue of workflows (the Queue Manager, since I haven't really gone through that one in full yet, which indicates that it's not yet done). But the TaskQuerier and Task Queue I'm pretty sure about, right?

          Show
          Chris A. Mattmann added a comment - haha you like that? Maybe I was using FILO prioritizer hahah. As for the queues, yeah one of them might be queue of workflows (the Queue Manager, since I haven't really gone through that one in full yet, which indicates that it's not yet done). But the TaskQuerier and Task Queue I'm pretty sure about, right?
          Hide
          Brian Foster added a comment -

          RE: arch figure ... t1, tN, t2, t3... lol... is that trying to show that is was sorted?... i haven't had a chance to look over all your changes to trunk workflow, but is there really 2 task queues? or is Queue Manager a queue of workflows?

          Show
          Brian Foster added a comment - RE: arch figure ... t1, tN, t2, t3... lol... is that trying to show that is was sorted?... i haven't had a chance to look over all your changes to trunk workflow, but is there really 2 task queues? or is Queue Manager a queue of workflows?
          Hide
          Chris A. Mattmann added a comment -

          arch figure

          Show
          Chris A. Mattmann added a comment - arch figure
          Hide
          Chris A. Mattmann added a comment -
          • this will be done by 0.4
          Show
          Chris A. Mattmann added a comment - this will be done by 0.4
          Hide
          Chris A. Mattmann added a comment -

          Note, a lot of this work exists already in the wengine-branch, but I'm filing it here so we can track progress amongst it, and the other issues I recently filed.

          Show
          Chris A. Mattmann added a comment - Note, a lot of this work exists already in the wengine-branch, but I'm filing it here so we can track progress amongst it, and the other issues I recently filed.

            People

            • Assignee:
              Chris A. Mattmann
              Reporter:
              Chris A. Mattmann
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development