Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-1976

Specifying coordinator input datasets in more logical ways

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • trunk
    • 4.3.0
    • coordinator
    • None

    Description

      All dataset instances specified as input to coordinator, currently work on AND logic i.e. ALL of them should be available for workflow to start. We should enhance this to include more logical ways of specifying availability criteria e.g.

      • OR between instances
      • minimum N out of K instances
      • delta datasets (process data incrementally)

      Use-cases for this:

      • Different datasets are BCP, and workflow can run with either, whichever arrives earlier.
      • Data is not guaranteed, and while $coord:latest allows skipping to available ones, workflow will never trigger unless mentioned number of instances are found.
      • Workflow is like a ‘refining’ algorithm which should run after minimum required datasets are ready, and should only process the delta for efficiency.

      This JIRA is to discuss the design and then the review the implementation for some or all of the above features.

      Attachments

        1. amend-OOZIE-1976-V2.patch
          23 kB
          Purshotam Shah
        2. amend-OOZIE-1976-V1.patch
          12 kB
          Purshotam Shah
        3. OOZIE-1976-V8.patch
          351 kB
          Purshotam Shah
        4. OOZIE-1976-V7.patch
          351 kB
          Purshotam Shah
        5. OOZIE-1976-V6.patch
          351 kB
          Purshotam Shah
        6. OOZIE-1976-V5.patch
          350 kB
          Purshotam Shah
        7. OOZIE-1976-V3.patch
          351 kB
          Purshotam Shah
        8. Input-check.docx
          20 kB
          Purshotam Shah
        9. OOZIE-1976-WIP.patch
          70 kB
          Mona Chitnis
        10. OOZIE-1976-rough-design-2.pdf
          136 kB
          Mona Chitnis
        11. OOZIE-1976-rough-design.pdf
          74 kB
          Mona Chitnis

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            puru Purshotam Shah
            chitnis Mona Chitnis
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment