Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25075 Build and test Spark against Scala 2.13
  3. SPARK-36636

SparkContextSuite random failure in Scala 2.13

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.2.0, 3.1.3, 3.0.4
    • Spark Core
    • None

    Description

      run

      build/mvn clean install -Pscala-2.13 -pl core -am

      or

      build/mvn clean install -Pscala-2.13 -pl core -am -Dtest=none -DwildcardSuites=org.apache.spark.SparkContextSuite
      

      Some cases may fail as follows:

       

      - SPARK-33084: Add jar support Ivy URI -- test param key case sensitive *** FAILED ***
        java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
      This stopped SparkContext was created at:
      
      
      org.apache.spark.SparkContextSuite.$anonfun$new$154(SparkContextSuite.scala:1155)
      org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      org.scalatest.Transformer.apply(Transformer.scala:22)
      org.scalatest.Transformer.apply(Transformer.scala:20)
      org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
      org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
      org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
      org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
      org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:62)
      org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
      org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
      org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:62)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
      org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      scala.collection.immutable.List.foreach(List.scala:333)
      
      
      The currently active SparkContext was created at:
      
      
      org.apache.spark.SparkContextSuite.$anonfun$new$154(SparkContextSuite.scala:1155)
      org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      org.scalatest.Transformer.apply(Transformer.scala:22)
      org.scalatest.Transformer.apply(Transformer.scala:20)
      org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
      org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
      org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
      org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
      org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:62)
      org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
      org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
      org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:62)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
      org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      scala.collection.immutable.List.foreach(List.scala:333)
        at org.apache.spark.SparkContext.assertNotStopped(SparkContext.scala:118)
        at org.apache.spark.SparkContext.getSchedulingMode(SparkContext.scala:1887)
        at org.apache.spark.SparkContext.postEnvironmentUpdate(SparkContext.scala:2575)
        at org.apache.spark.SparkContext.addJar(SparkContext.scala:2008)
        at org.apache.spark.SparkContext.addJar(SparkContext.scala:1928)
        at org.apache.spark.SparkContextSuite.$anonfun$new$154(SparkContextSuite.scala:1156)
        at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
        at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
        at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
        at org.scalatest.Transformer.apply(Transformer.scala:22)
        ...
      - SPARK-33084: Add jar support Ivy URI -- test transitive value case insensitive *** FAILED ***
        org.apache.spark.SparkException: Only one SparkContext should be running in this JVM (see SPARK-2243).The currently running SparkContext was created at:
      org.apache.spark.SparkContextSuite.$anonfun$new$154(SparkContextSuite.scala:1155)
      org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      org.scalatest.Transformer.apply(Transformer.scala:22)
      org.scalatest.Transformer.apply(Transformer.scala:20)
      org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
      org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
      org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
      org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
      org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:62)
      org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
      org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
      org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:62)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
      org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      scala.collection.immutable.List.foreach(List.scala:333)
        at org.apache.spark.SparkContext$.$anonfun$assertNoOtherContextIsRunning$2(SparkContext.scala:2647)
        at scala.Option.foreach(Option.scala:437)
        at org.apache.spark.SparkContext$.assertNoOtherContextIsRunning(SparkContext.scala:2644)
        at org.apache.spark.SparkContext$.markPartiallyConstructed(SparkContext.scala:2734)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:95)
        at org.apache.spark.SparkContextSuite.$anonfun$new$159(SparkContextSuite.scala:1166)
        at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
        at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
        at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
        at org.scalatest.Transformer.apply(Transformer.scala:22)
        ...
      - SPARK-34346: hadoop configuration priority for spark/hive/hadoop configs *** FAILED ***
        org.apache.spark.SparkException: Only one SparkContext should be running in this JVM (see SPARK-2243).The currently running SparkContext was created at:
      org.apache.spark.SparkContextSuite.$anonfun$new$154(SparkContextSuite.scala:1155)
      org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      org.scalatest.Transformer.apply(Transformer.scala:22)
      org.scalatest.Transformer.apply(Transformer.scala:20)
      org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
      org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
      org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
      org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
      org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
      org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:62)
      org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
      org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
      org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:62)
      org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
      org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      scala.collection.immutable.List.foreach(List.scala:333)
        at org.apache.spark.SparkContext$.$anonfun$assertNoOtherContextIsRunning$2(SparkContext.scala:2647)
        at scala.Option.foreach(Option.scala:437)
        at org.apache.spark.SparkContext$.assertNoOtherContextIsRunning(SparkContext.scala:2644)
        at org.apache.spark.SparkContext$.markPartiallyConstructed(SparkContext.scala:2734)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:95)
        at org.apache.spark.SparkContextSuite.$anonfun$new$164(SparkContextSuite.scala:1192)
        at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
        at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
        at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
        at org.scalatest.Transformer.apply(Transformer.scala:22)
        ...
      - SPARK-34225: addFile/addJar shouldn't further encode URI if a URI form string is passed *** FAILED ***
        java.lang.NullPointerException:
        at org.apache.spark.SparkContextSuite.$anonfun$new$166(SparkContextSuite.scala:1245)
        at org.apache.spark.SparkContextSuite.$anonfun$new$166$adapted(SparkContextSuite.scala:1211)
        at org.apache.spark.SparkFunSuite.withTempDir(SparkFunSuite.scala:208)
        at org.apache.spark.SparkContextSuite.$anonfun$new$165(SparkContextSuite.scala:1211)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
        at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
        at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
        at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
        at org.scalatest.Transformer.apply(Transformer.scala:22)
        at org.scalatest.Transformer.apply(Transformer.scala:20)
        ...
      

       but which case fails has a certain randomness

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            LuciferYang Yang Jie
            LuciferYang Yang Jie
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment