[SPARK-33986] Spark handle always return LOST status in standalone cluster mode with Spark launcher

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.4.4
• Component/s: Spark Submit

Environment

apache hadoop 2.6.5

apache spark 2.4.4

• Assignee: Unassigned
• Reporter: ZhongyuWang
• Created: 04/Jan/21 10:59
• Updated: 12/Dec/22 18:10
• Resolved: 06/Jan/21 02:58

Description

I can use it to submit spark app successfully in standalone client/yarn client/yarn cluster mode，and get correct app status, but when i submit spark app in standalone cluster mode, Spark handle always return LOST status(once) and app running stablely until FINISHED( handle wasn't get any state change infomation).  I noticed when I submited app from code, after a while, the SparkSubmit process was suddenly stopped. I checked sparkSubmit log(launcher redirect log) doesn't have any useful information.

this is my pseudo code,

SparkAppHandle handle = launcher.startApplication(new SparkAppHandle.Listener() {
    @Override
    public void stateChanged(SparkAppHandle handle) {
        stateChangedHandle(handle.getAppId(), jobId, code, execId, handle.getState(), driverInfo, request, infoLog, errorLog);
    }
    @Override
    public void infoChanged(SparkAppHandle handle) {
        stateChangedHandle(handle.getAppId(), jobId, code, execId, handle.getState(), driverInfo, request, infoLog, errorLog);
    }
});

any idea ? thx