[FLINK-14268] YARN AM endless restarts when using wrong checkpoint path or wrong checkpoint

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Abandoned
• Affects Version/s: 1.7.2
• Component/s: Deployment / YARN

Environment

Flink: 1.7.2

Deloyment: YARN Per Job

YARN:2.7.2

State backend:FSStateBackend with HDFS 

 

• Assignee: Unassigned
• Reporter: Lsw_aka_laplace
• Created: 29/Sep/19 04:46
• Updated: 28/Aug/20 10:12
• Resolved: 28/Aug/20 10:12

Description

I tried to start a  streaming task and restore from checkpoint which it was stored in HDFS. 

I set a wrong checkpoint path and sth unexpected happened: YARN AM restarted again and again.  Since we have already set some restart strategy to prevent endless restart, it should have been restarted with limited times.

Since we made sure that restart strategy works, we dived into source code and did some change mainly in ClusterEntrypoint.

 

//代码占位符
//before 
@Override
public void onFatalError(Throwable exception) {
   LOG.error("Fatal error occurred in the cluster entrypoint.", exception);

   System.exit(RUNTIME_FAILURE_RETURN_CODE);
}


//after 
@Override
public void onFatalError(Throwable exception) {
   LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
 if(ExceptionUtils.findThrowable(exception,PerJobFatalException.class).isPresent()){
//PerJobFatalException is the FLAG 
//在perjob模式有些致命的异常出现，am会一直重启，不能失败掉
      LOG.error("perjob fatal error");
      System.exit(STARTUP_FAILURE_RETURN_CODE);
   }
   System.exit(RUNTIME_FAILURE_RETURN_CODE);
}

 We forced to make the FAILURE_RETURN_CODE as STARTUP_FAILURE_RETURN_CODE rather than RUNTIME_FAILURE_RETURN_CODE in some condition and it DID WORK.

 

 

After discussing with Tison,  I knew that FAILURE_RETURN_CODE seems only to be used to debug, so I submitted this issue and look forward to ANY solution~