[GOBBLIN-1954] Update FlowSpec Safely to Avoid Deserialization and Compilation Failure

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: gobblin-service
• Assignee: Abhishek Tiwari
• Reporter: Urmi Mustafi
• Created: 09/Nov/23 01:24
• Updated: 09/Nov/23 01:27

Description

When copying a flowSpec to add flowExecutionId into its config, all components need to be carefully duplicated to ensure compilation, serialization and deserialization are successful. Some pieces can easily be missed so instead, we add the property to the Properties object of the FlowSpec which is mutable. Note that we do not update the Config object to avoid large GC overhead from deleting and creating a new Config object. This test also ensures that the DagManager thread will not silently die during leadership changes by catching all types of Exceptions.