[GOBBLIN-2107] Delete adhoc flowSpecs from flowCatalog

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: gobblin-service
• Assignee: Abhishek Tiwari
• Reporter: Urmi Mustafi
• Created: 09/Jul/24 00:28
• Updated: 10/Jul/24 19:07

Description

Delete adhoc flowSpecs from flowCatalog to avoid build up of adhoc flowSpecs in catalog even when job compilation fails by adding the deletion in a finally block in non-MA scheduler case. 

1. Previous PR: https://github.com/apache/gobblin/pull/3944/files causes a bug where adhoc flows are not deleted when multi-active scheduler is enabled. This PR re-adds deletion of flowSpec to the dagManager to be called by the active host for the MA scheduler case (there will only be one active host if dagManager is enabled otherwise code will use dagProcessingEngine). 
2. It also ensures quota is released and failed flow compilation event sent by the FlowCompilationValidationHelper in all compilation failure cases. This was being incorrectly handled before.