Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26501

Unexpected overriden of exitFn in SparkSubmitSuite

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.2, 2.4.0
    • Fix Version/s: 2.4.1, 3.0.0
    • Component/s: Deploy, Spark Core
    • Labels:
      None

      Description

      When I run SparkSubmitSuite of spark2.3.2 in intellij IDE, I found that some tests cannot pass when I run them one by one, but they passed when the whole SparkSubmitSuite was run.

      Failed tests when ran seperately:

       

      test("SPARK_CONF_DIR overrides spark-defaults.conf") {
        forConfDir(Map("spark.executor.memory" -> "2.3g")) { path =>
          val unusedJar = TestUtils.createJarWithClasses(Seq.empty)
          val args = Seq(
            "--class", SimpleApplicationTest.getClass.getName.stripSuffix("$"),
            "--name", "testApp",
            "--master", "local",
            unusedJar.toString)
          val appArgs = new SparkSubmitArguments(args, Map("SPARK_CONF_DIR" -> path))
          assert(appArgs.defaultPropertiesFile != null)
          assert(appArgs.defaultPropertiesFile.startsWith(path))
          assert(appArgs.propertiesFile == null)
          appArgs.executorMemory should be ("2.3g")
        }
      }
      

      Failure reason:

      Error: Executor Memory cores must be a positive number
      Run with --help for usage help or --verbose for debug output
      

       

      After carefully checked the code, I found the exitFn of SparkSubmit is overrided by front tests in testPrematrueExit call.

      Although the above test was fixed by SPARK-22941, but the overriden of exitFn might cause other problems in the future.

       

        Attachments

          Activity

            People

            • Assignee:
              liupengcheng liupengcheng
              Reporter:
              liupengcheng liupengcheng
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: