[GOBBLIN-1948] Use Consist flowExecutionId Across Participants

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: gobblin-service
• Assignee: Abhishek Tiwari
• Reporter: Urmi Mustafi
• Created: 02/Nov/23 18:53
• Updated: 03/Nov/23 17:51

Description

A consistent flowExecutionId is used during lease arbitration between hosts and that id is available to the DagActionStoreChangeMonitor but not used when passing launch events to the Orchestrator for recompilation and eventual execution. The unintended consequence of this is that a different flowExecutionId will be used across each participant when the flow is recompiled before passing to the DagManager. This messes up the job status of the most recent flow execution of each flow as it appears there are N flow execution Ids, 1 for each of the N hosts. Only one will be executed but the other N-1 are stuck in "compiling" job status state.