[GOBBLIN-1352] Ensure Helix workflows are cleaned up on cluster start up in centralized mode

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.16.0
• Fix Version/s: 0.17.0
• Component/s: gobblin-cluster
• Assignee: Hung Tran
• Reporter: Sudarshan Vasudevan
• Created: 06/Jan/21 21:46
• Updated: 14/Jun/23 04:13

Description

As part of the changes introduced in GOBBLIN-1319, the Helix workflow names were modified to include a special prefix. However, this prefix is intended for use in distributed mode. As a consequence of this change, the workflows are no longer being cleaned up on cluster start up leading to a leak in Znodes. Further, since the workflows are left in the START state, the Yarn autoscaling manager requests more containers than needed.