[GOBBLIN-1984] Add Consensus FlowExecutionId to FlowSpec used for Compilation and Tracking Dag

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: gobblin-service
• Assignee: Abhishek Tiwari
• Reporter: Urmi Mustafi
• Created: 12/Jan/24 00:03
• Updated: 16/Jan/24 18:05

Description

In a previous change, we noticed the compiled flowExecutionId was different from the one during execution and tried to fix it by updating the flowMetadata used to send GTE relating to compilation. We succeeded in updating the compilation event to use the flowExecutionId but did not update the FlowSpec used by Compiler to create the Dag which is used during the execution phase of flow management. The compiler retrieves a flowExecutionId if one exists in the flowSpec (which it does for adhoc flows), otherwise generates a new one (code We do need to modify the FlowSpec then with the consensus flowExecutionId before forwarding it to the DagManager.