[SPARK-33490] calling system.exit(-1) from a future does not result in the driver reflecting the fact that the spark app failed.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.4.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Srimantha Rajapaksa
• Created: 19/Nov/20 14:25
• Updated: 01/Mar/21 14:50
• Resolved: 01/Mar/21 14:50

Description

Hi Guys,

when calling system. exit(-1) from within a code block defined for a future observed that the vm does not exit immediately as you would expect a java app to do. in fact it seems to exit the future thread and the diver program carries on as normal. the status of the app when it exits is also misleading. It says FINISHED instead of FAILED. I have attached code sample see below) that replicate this issue. is this a spark bug or a "feature"   any thoughts?

Note: I have been able to work around this by not calling system .exit(-1) from within a future but this was definitely a gotcha that I thought worthwhile raising..  

// code placeholder
package com.spark.example

import com.typesafe.config.Config
import com.typesafe.scalalogging.Logger
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession

import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration.DurationInt
import scala.concurrent.{Await, Future}
import scala.util.{Failure, Success, Try}

object SystemExitTestApp extends App {
  val appName = "NumberCountSparkApp"

  val logger = Logger(getClass)
  val sparkMaster = "spark://adl-matspm01:7077"

  val ss = SparkSession.builder()
    .master(sparkMaster)
    .appName(appName).getOrCreate()


  logger.info(s"Starting test. thread id ${Thread.currentThread().getId}")

  val data = Seq(1, 2, 3, 4)
  val rdd: RDD[Int] = ss.sparkContext.parallelize(data)
  val future = Future {
    logger.info(s"inside future thread id ${Thread.currentThread().getId}")
    Try {
      throw new Exception("My dummy exception")
      rdd.count()
    } match {
      case Success(_) => {
        (true, 1)
      }
      case Failure(exception) => {
        logger.info(s"inside future. Exception ${exception} thread id ${Thread.currentThread().getId}")
        (false, 0)
      }
    }
  }
  var result: Boolean = true
  future onComplete {
    case Success(x) => {
      logger.info(s"success ${x}")
      if (!x._1) {
        logger.info(s"exiting in the success block of on complete. thread id ${Thread.currentThread().getId}")
        result = x._1
        ss.close()
        sys.exit(-1)
      } else {
        logger.info(s"not exiting in the success block of on complete. thread id ${Thread.currentThread().getId}")
        result = x._1
      }
    }
    case Failure(exception) => {
      logger.info(s"exiting in the failure block of on complete as exception occurred ${exception}. thread id ${Thread.currentThread().getId}")
      result = false
      sys.exit(-1)
    }
  }
  Await.result(future, 3 hours)
  logger.info(s"should not see this really if we do lets use the result outside. thread id ${Thread.currentThread().getId}")
  if (!result) {
    logger.info(s" inside main block should not see this really. thread id ${Thread.currentThread().getId}")
    //sys.exit(-1)
  }


}