Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25075 Build and test Spark against Scala 2.13
  3. SPARK-36636

SparkContextSuite random failure in Scala 2.13

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.2.0, 3.1.3, 3.0.4
    • Component/s: Spark Core
    • Labels:
      None

      Description

      run

      build/mvn clean install -Pscala-2.13 -pl core -am

      or

      build/mvn clean install -Pscala-2.13 -pl core -am -Dtest=none -DwildcardSuites=org.apache.spark.SparkContextSuite
      

      Some cases may fail as follows:

       

      - SPARK-33084: Add jar support Ivy URI -- test param key case sensitive *** FAILED ***
        java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
      This stopped SparkContext was created at:
      
      
      org.apache.spark.SparkContextSuite.$anonfun$new$154(SparkContextSuite.scala:1155)
      org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      org.scalatest.Transformer.apply(Transformer.scala:22)
      org.scalatest.Transformer.apply(Transformer.scala:20)
      org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
      org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
      org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
      org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
      org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:62)
      org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
      org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
      org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:62)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
      org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      scala.collection.immutable.List.foreach(List.scala:333)
      
      
      The currently active SparkContext was created at:
      
      
      org.apache.spark.SparkContextSuite.$anonfun$new$154(SparkContextSuite.scala:1155)
      org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      org.scalatest.Transformer.apply(Transformer.scala:22)
      org.scalatest.Transformer.apply(Transformer.scala:20)
      org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
      org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
      org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
      org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
      org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:62)
      org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
      org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
      org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:62)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
      org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      scala.collection.immutable.List.foreach(List.scala:333)
        at org.apache.spark.SparkContext.assertNotStopped(SparkContext.scala:118)
        at org.apache.spark.SparkContext.getSchedulingMode(SparkContext.scala:1887)
        at org.apache.spark.SparkContext.postEnvironmentUpdate(SparkContext.scala:2575)
        at org.apache.spark.SparkContext.addJar(SparkContext.scala:2008)
        at org.apache.spark.SparkContext.addJar(SparkContext.scala:1928)
        at org.apache.spark.SparkContextSuite.$anonfun$new$154(SparkContextSuite.scala:1156)
        at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
        at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
        at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
        at org.scalatest.Transformer.apply(Transformer.scala:22)
        ...
      - SPARK-33084: Add jar support Ivy URI -- test transitive value case insensitive *** FAILED ***
        org.apache.spark.SparkException: Only one SparkContext should be running in this JVM (see SPARK-2243).The currently running SparkContext was created at:
      org.apache.spark.SparkContextSuite.$anonfun$new$154(SparkContextSuite.scala:1155)
      org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      org.scalatest.Transformer.apply(Transformer.scala:22)
      org.scalatest.Transformer.apply(Transformer.scala:20)
      org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
      org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
      org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
      org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
      org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:62)
      org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
      org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
      org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:62)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
      org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      scala.collection.immutable.List.foreach(List.scala:333)
        at org.apache.spark.SparkContext$.$anonfun$assertNoOtherContextIsRunning$2(SparkContext.scala:2647)
        at scala.Option.foreach(Option.scala:437)
        at org.apache.spark.SparkContext$.assertNoOtherContextIsRunning(SparkContext.scala:2644)
        at org.apache.spark.SparkContext$.markPartiallyConstructed(SparkContext.scala:2734)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:95)
        at org.apache.spark.SparkContextSuite.$anonfun$new$159(SparkContextSuite.scala:1166)
        at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
        at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
        at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
        at org.scalatest.Transformer.apply(Transformer.scala:22)
        ...
      - SPARK-34346: hadoop configuration priority for spark/hive/hadoop configs *** FAILED ***
        org.apache.spark.SparkException: Only one SparkContext should be running in this JVM (see SPARK-2243).The currently running SparkContext was created at:
      org.apache.spark.SparkContextSuite.$anonfun$new$154(SparkContextSuite.scala:1155)
      org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      org.scalatest.Transformer.apply(Transformer.scala:22)
      org.scalatest.Transformer.apply(Transformer.scala:20)
      org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
      org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
      org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
      org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
      org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:62)
      org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
      org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
      org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:62)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
      org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      scala.collection.immutable.List.foreach(List.scala:333)
        at org.apache.spark.SparkContext$.$anonfun$assertNoOtherContextIsRunning$2(SparkContext.scala:2647)
        at scala.Option.foreach(Option.scala:437)
        at org.apache.spark.SparkContext$.assertNoOtherContextIsRunning(SparkContext.scala:2644)
        at org.apache.spark.SparkContext$.markPartiallyConstructed(SparkContext.scala:2734)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:95)
        at org.apache.spark.SparkContextSuite.$anonfun$new$164(SparkContextSuite.scala:1192)
        at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
        at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
        at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
        at org.scalatest.Transformer.apply(Transformer.scala:22)
        ...
      - SPARK-34225: addFile/addJar shouldn't further encode URI if a URI form string is passed *** FAILED ***
        java.lang.NullPointerException:
        at org.apache.spark.SparkContextSuite.$anonfun$new$166(SparkContextSuite.scala:1245)
        at org.apache.spark.SparkContextSuite.$anonfun$new$166$adapted(SparkContextSuite.scala:1211)
        at org.apache.spark.SparkFunSuite.withTempDir(SparkFunSuite.scala:208)
        at org.apache.spark.SparkContextSuite.$anonfun$new$165(SparkContextSuite.scala:1211)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
        at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
        at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
        at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
        at org.scalatest.Transformer.apply(Transformer.scala:22)
        at org.scalatest.Transformer.apply(Transformer.scala:20)
        ...
      

       but which case fails has a certain randomness

        Attachments

        1. image-2021-09-11-00-29-21-168.png
          179 kB
          Yang Jie
        2. image-2021-09-11-00-30-20-237.png
          179 kB
          Yang Jie
        3. image-2021-09-11-00-39-43-752.png
          114 kB
          Yang Jie

          Activity

            People

            • Assignee:
              LuciferYang Yang Jie
              Reporter:
              LuciferYang Yang Jie
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: