Uploaded image for project: 'Apache Gobblin'
  1. Apache Gobblin
  2. GOBBLIN-474

Add end-of-datasets marker to workunit correctly to avoid an additional run when end of datasets is reached.

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.15.0
    • Component/s: gobblin-compliance
    • Labels:
      None

      Description

      When the number of maximum workunits per runĀ setting in LoopingDatasetFinderSource equals the number of remaining datasets to process before the end of the loop, we do not place the "End-of-Dataset" marker in the last workunit. The subsequent run turns out to be essentially a no-op and is avoidable by correctly detecting the end-of-datasets.

        Attachments

          Activity

            People

            • Assignee:
              sv2000 Sudarshan Vasudevan
              Reporter:
              sv2000 Sudarshan Vasudevan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: