[GOBBLIN-1865] Fix bug where overriding job execution ids cause issue with earlystop jobs and job tracking

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: gobblin-cluster
• Assignee: Hung Tran
• Reporter: William Lo
• Created: 02/Aug/23 20:05
• Updated: 02/Aug/23 20:31

Description

With "gobblin.cluster.job.useGeneratedJobIds" configuration, jobs with that prefix should be using the system timestamp of Gobblin cluster instead of a provided flow execution ID.

Instead of this, it is more consistent to append flowExecutionId to a jobName then append a timestamp on top of that, so that all earlystop jobs relating to a flow execution can be tracked.

Now jobNames should have the following structure:
job_ActualJob<jobName><flowExecutionId><timestamp>

The timestamp is needed so that Helix can run concurrent jobs given a job ID.