[FLINK-24063] Reconsider the behavior of ClusterEntrypoint#startCluster failure handler

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Runtime / Coordination
• Assignee: Unassigned
• Reporter: WenJun Min
• Created: 31/Aug/21 05:38
• Updated: 03/Sep/21 05:35

Description

If the job runCluster failed, it will trigger the STOP_APPLICATION behavior. But if we consider a case like that:

1. A job have running for a long time
2. Then the JobManager encounter a fatal error like the network problem, which may let the jobManager process down
3. Then a new process will be started by the resource framework like yarn or kubernetes. But it will failed at the ClusterEntrypoint#startCluster due to the same network problem. 
4. Then the job turn into the FAILED status.

 

This means  a streaming job will no longer run due to some fatal error, this is somehow fragile. I think we should give some retry mechanism to prevent the job fast fail twice ,so that deal with some external error which may keep for a period of time.