[GOBBLIN-474] Add end-of-datasets marker to workunit correctly to avoid an additional run when end of datasets is reached.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.15.0
• Fix Version/s: 0.17.0
• Component/s: gobblin-compliance
• Assignee: Sudarshan Vasudevan
• Reporter: Sudarshan Vasudevan
• Created: 24/Apr/18 20:48
• Updated: 14/Jun/23 04:13

Description

When the number of maximum workunits per run setting in LoopingDatasetFinderSource equals the number of remaining datasets to process before the end of the loop, we do not place the "End-of-Dataset" marker in the last workunit. The subsequent run turns out to be essentially a no-op and is avoidable by correctly detecting the end-of-datasets.