Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33986

Spark handle always return LOST status in standalone cluster mode with Spark launcher

    XMLWordPrintableJSON

    Details

    • Type: Question
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 2.4.4
    • Fix Version/s: None
    • Component/s: Spark Submit
    • Labels:
      None
    • Environment:

      apache hadoop 2.6.5

      apache spark 2.4.4

      Description

      I can use it to submit spark app successfully in standalone client/yarn client/yarn cluster mode,and get correct app status, but when i submit spark app in standalone cluster mode, Spark handle always return LOST status(once) and app running stablely until FINISHED( handle wasn't get any state change infomation).  I noticed when I submited app from code, after a while, the SparkSubmit process was suddenly stopped. I checked sparkSubmit log(launcher redirect log) doesn't have any useful information.

      this is my pseudo code,

      SparkAppHandle handle = launcher.startApplication(new SparkAppHandle.Listener() {
          @Override
          public void stateChanged(SparkAppHandle handle) {
              stateChangedHandle(handle.getAppId(), jobId, code, execId, handle.getState(), driverInfo, request, infoLog, errorLog);
          }
          @Override
          public void infoChanged(SparkAppHandle handle) {
              stateChangedHandle(handle.getAppId(), jobId, code, execId, handle.getState(), driverInfo, request, infoLog, errorLog);
          }
      });

      any idea ? thx

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ZhongyuWang ZhongyuWang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: