[GOBBLIN-798] Clean up workflows from Helix when the Gobblin application master starts

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15.0
• Assignee: Hung Tran
• Reporter: Hung Tran
• Created: 06/Jun/19 23:40
• Updated: 10/Jun/19 17:05
• Resolved: 10/Jun/19 17:05

Description

If the application master aborts a new one may be spawned by YARN. The second application master will resubmit the jobs. This results in duplicate jobs in Helix and multiple instances of the job may run, resulting in duplicate data.

The Gobblin application master should clean up all workflows on startup to avoid executing multiple instances of a job.

 

Issue Links

• links to: GitHub Pull Request #2665