[GOBBLIN-1947] Send WorkUnitChangeEvent when helix task consistently fail

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: gobblin-cluster
• Assignee: Hung Tran
• Reporter: Hanghang Liu
• Created: 02/Nov/23 04:30
• Updated: 04/Jan/24 21:38

Description

When YarnAutoScalingManager detect helix task consistently fail, give an option to send WorkUnitChangeEvent to let GobblinHelixJobLauncher handle the event and split the work unit during runtime. This can help resolving consistent failing containers issue(like OOM) during runtime instead of relying on replaner to restart the whole pipeline