[GOBBLIN-616] Add ability to fork jobs when concatenating Dags.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.14.0
• Component/s: gobblin-service
• Assignee: Sudarshan Vasudevan
• Reporter: Sudarshan Vasudevan
• Created: 19/Oct/18 21:17
• Updated: 22/Oct/18 20:43
• Resolved: 22/Oct/18 20:43

Description

We add the ability to fork jobs when concatenating 2 dags, as part of constructing a multi-hop path. An example use case is to allow retention jobs on each hop of the path to run in parallel to the distcp jobs along a multi-hop path, thus preventing the data copy on a subsequent hop from being blocked on a retention job on a previous hop.