Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17742

Spark Launcher does not get failed state in Listener

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.3.0
    • Component/s: Spark Submit
    • Labels:
      None

      Description

      I tried to launch an application using the below code. This is dummy code to reproduce the problem. I tried exiting spark with status -1, throwing an exception etc. but in no case did the listener give me failed status. But if a spark job returns -1 or throws an exception from the main method it should be considered as a failure.

      package com.example;
      
      import org.apache.spark.launcher.SparkAppHandle;
      import org.apache.spark.launcher.SparkLauncher;
      
      import java.io.IOException;
      
      public class Main2 {
      
          public static void main(String[] args) throws IOException, InterruptedException {
              SparkLauncher launcher = new SparkLauncher()
                      .setSparkHome("/opt/spark2")
                      .setAppResource("/home/aseem/projects/testsparkjob/build/libs/testsparkjob-1.0-SNAPSHOT.jar")
                      .setMainClass("com.example.Main")
                      .setMaster("local[2]");
      
              launcher.startApplication(new MyListener());
      
              Thread.sleep(1000 * 60);
          }
      
      }
      
      class MyListener implements SparkAppHandle.Listener {
      
          @Override
          public void stateChanged(SparkAppHandle handle) {
      
              System.out.println("state changed " + handle.getState());
          }
      
          @Override
          public void infoChanged(SparkAppHandle handle) {
              System.out.println("info changed " + handle.getState());
          }
      }
      

      The spark job is

      package com.example;
      
      import org.apache.spark.sql.SparkSession;
      import java.io.IOException;
      
      public class Main {
      
          public static void main(String[] args) throws IOException {
              SparkSession sparkSession = SparkSession
                      .builder()
                      .appName("" + System.currentTimeMillis())
                      .getOrCreate();
      
      
              try {
                  for (int i = 0; i < 15; i++) {
                      Thread.sleep(1000);
                      System.out.println("sleeping 1");
                  }
              } catch (InterruptedException e) {
                  e.printStackTrace();
              }
      //        sparkSession.stop();
      
              System.exit(-1);
          }
      
      }
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Vanzin
                Reporter:
                anshbansal Aseem Bansal
              • Votes:
                3 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: