Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25075

Build and test Spark against Scala 2.13

    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      This umbrella JIRA tracks the requirements for building and testing Spark against the current Scala 2.13 milestone.

        Attachments

        1.
        Use scala-xml explicitly Sub-task Resolved Unassigned
        2.
        Remove usage of Traversable Sub-task Resolved Sean R. Owen
        3.
        Avoid use of Scala collection classes that are removed in 2.13 Sub-task Resolved Sean R. Owen
        4.
        Remove usage of TraversableOnce Sub-task Resolved Sean R. Owen
        5.
        Upgrade breeze to 1.0 Sub-task Resolved Sean R. Owen
        6.
        Update scalatest, scalacheck, scopt, scala-parser-combinators, clapper classutil Sub-task Resolved Sean R. Owen
        7.
        Update twitter chill for 2.13 Sub-task Resolved Sean R. Owen
        8.
        Remove use of procedure-like method definition for 2.13 Sub-task Resolved Sean R. Owen
        9.
        Fix internal usages of mutable collection for Seq in 2.13 Sub-task Resolved Sean R. Owen
        10.
        Move scalafmt to Scala 2.12 profile; bump to 0.12 Sub-task Resolved Sean R. Owen
        11.
        Update Kafka to 2.4.0 for Scala 2.13 Sub-task Resolved Jungtaek Lim
        12.
        Use scala-parallel-collections library in 2.13 Sub-task Resolved Sean R. Owen
        13.
        Update REPL for 2.13 Sub-task Resolved Karol Chmist
        14.
        Remove use of deprecated symbol literal " 'name " syntax in favor Symbol("name") Sub-task Resolved Sean R. Owen
        15.
        Replace ambiguous varargs call parallelize(Array) with parallelize(Seq) Sub-task Resolved Sean R. Owen
        16.
        Remove use of Unit companion object in favor of () Sub-task Resolved Sean R. Owen
        17.
        Remove use of TraversableLike Sub-task Resolved Sean R. Owen
        18.
        Replace Array.deep comparisons with Array.sameElements Sub-task Resolved Sean R. Owen
        19.
        Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 Sub-task Resolved Dongjoon Hyun
        20.
        Replace Ordering.Double with Ordering.Double.TotalOrdering Sub-task Resolved Sean R. Owen
        21.
        Remove deprecated SparkConf.setAll(Traversable) Sub-task Resolved Sean R. Owen
        22.
        Use DoubleAsIfIntegral / FloatAsIfIntegral workalike in 2.13 Sub-task Resolved Sean R. Owen
        23.
        Change classes extending scala collection classes to work with 2.13 Sub-task Resolved Sean R. Owen
        24.
        No parens for calls to BigDecimal.longValue() etc. Sub-task Resolved Sean R. Owen
        25.
        Scala 2.13 compile errors from Hadoop LocalFileSystem subclasses Sub-task Closed Unassigned
        26.
        Resolve ambiguous overload from RegexParsers in 2.13 Sub-task Resolved Sean R. Owen
        27.
        Resolve Array + reference type compile problems in 2.13, with sc.parallelize Sub-task Resolved Sean R. Owen
        28.
        Some imports, function need more explicit resolution in 2.13 Sub-task Resolved Sean R. Owen
        29.
        Upgrade Genjavadoc to 0.15 Sub-task Resolved Kousuke Saruta
        30.
        Iterator doesn't refer outer identifier named "iterator" properly in Scala 2.13 Sub-task Resolved Kousuke Saruta
        31.
        Get tests working for Scala 2.13 build Sub-task Resolved Sean R. Owen
        32.
        Upgrade to scalatest 3.2.0 for Scala 2.13.3 compatibility Sub-task Resolved Sean R. Owen
        33.
        Upgrade Genjavadoc to 0.16 Sub-task Resolved Dongjoon Hyun
        34.
        Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts Sub-task Resolved Dongjoon Hyun
        35.
        Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal Sub-task Resolved Dongjoon Hyun
        36.
        Use HashMap.withDefaultValue in RDDSuite Sub-task Resolved Dongjoon Hyun
        37.
        Make BlockManagerSuite robust from Scala object size difference Sub-task Resolved Dongjoon Hyun
        38.
        Update json4s for Scala 2.13 Sub-task Resolved Dongjoon Hyun
        39.
        Fix TaskSetManagerSuite by hiding `org.apache.spark.FakeSchedulerBackend` Sub-task Resolved Dongjoon Hyun
        40.
        Fix testCommandAvailable to use POSIX compatible `command -v` Sub-task Resolved Hyukjin Kwon
        41.
        Fix HistoryServerSuite in Scala 2.13 Sub-task Resolved Unassigned
        42.
        Use === instead IndexSeqView Sub-task Resolved Dongjoon Hyun
        43.
        ResourceAllocator.availableAddrs should be deterministic Sub-task Resolved Dongjoon Hyun
        44.
        JsonProtocol.accumulablesToJson should be deterministic Sub-task Resolved Dongjoon Hyun
        45.
        Remove javax.ws.rs.NotFoundException from `import` in StagesResource/OneApplicationResource Sub-task Resolved Dongjoon Hyun
        46.
        Pass all `core` module UTs in Scala 2.13 Sub-task Resolved Dongjoon Hyun
        47.
        Let sql/catalyst module tests pass for Scala 2.13 Sub-task Resolved Yang Jie
        48.
        Fix CaseInsensitiveMap in Scala 2.13 Sub-task Resolved Karol Chmist
        49.
        Pass all `sql/core` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        50.
        Ignore deprecation warnings for build with Scala-2.13 and sbt Sub-task Resolved Kousuke Saruta
        51.
        Append toMap to Map#filterKeys if the result of filter is concatenated with another Map for Scala 2.13 Sub-task Resolved Kousuke Saruta
        52.
        Fix code which causes error when build with sbt and Scala 2.13 Sub-task Resolved Kousuke Saruta
        53.
        Pass all `sql/hive` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        54.
        Pass all `sql/hive-thriftserver` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        55.
        Add Scala 2.13 build test in GitHub Action Sub-task Resolved Dongjoon Hyun
        56.
        Pass all `external/avro` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        57.
        Pass all `streaming` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        58.
        Pass all `mllib` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        59.
        Pass all `mesos` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        60.
        Add a Jenkins build and test job for Scala 2.13 Sub-task Resolved Shane Knapp
        61.
        Update how to build doc for Scala 2.13 with sbt Sub-task Resolved Kousuke Saruta
        62.
        Fix SparkBuild.scala to recognize build settings for Scala 2.13 Sub-task Resolved Kousuke Saruta
        63.
        Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job Sub-task Resolved Kousuke Saruta
        64.
        Rename classOf[Seq] to classOf[scala.collection.Seq] in PostgresIntegrationSuite for Scala 2.13 Sub-task Resolved Kousuke Saruta
        65.
        Fix procedure-like declaration compilation warning in Scala 2.13 Sub-task Resolved Yang Jie
        66.
        Upgrade to Scala 2.13.4 Sub-task Resolved Dongjoon Hyun
        67.
        Upgrade Scala 2.13 from 2.13.3 to 2.13.4 Sub-task Closed Unassigned
        68.
        Change `InMemoryTable` not to use Tuple.hashCode for `BucketTransform` Sub-task Resolved Dongjoon Hyun
        69.
        Suppress unimportant compilation warnings in Scala 2.13 Sub-task Resolved Yang Jie
        70.
        Word2VecSuite failed for Scala 2.13 Sub-task Resolved koert kuipers
        71.
        ExpressionEncoderSuite failed in spark-branch-3.1-test-maven-hadoop-*-jdk-*-scala-2.13 Sub-task Resolved Yang Jie
        72.
        HadoopVersionInfoSuite faild when maven test in Scala 2.13 Sub-task Closed Unassigned
        73.
        Java UT failed when testing sql/hive module independently in Scala 2.13 Sub-task Resolved Yang Jie
        74.
        Add Scala 2.13 packaging and publishing Sub-task Resolved Dongjoon Hyun
        75.
        Upgrade Scala to avoid Scala's bug about illegal cyclic reference Sub-task Closed Unassigned
        76.
        Replace symbol literals with Symbol constructor invocations to comply with Scala 2.13 Sub-task In Progress Kousuke Saruta
        77.
        Upgrade Scala to 2.13.5 Sub-task Resolved Dongjoon Hyun
        78.
        Update sbt version to 1.4.9 Sub-task Resolved William Hyun
        79.
        Spark artefacts built against Scala 2.13 incorrectly depend on Scala 2.12 Sub-task Resolved Sean R. Owen
        80.
        Jekyll fails to generate Scala API docs for Scala 2.13 Sub-task Resolved Kousuke Saruta
        81.
        Upgrade SBT to 1.5.0 Sub-task Resolved Dongjoon Hyun
        82.
        Suppress `symbol literal is deprecated` compilation warnings in Scala 2.13 Sub-task In Progress Unassigned
        83.
        Upgrade GenJavadoc to 0.17 Sub-task Resolved Kousuke Saruta
        84.
        Many test cases failed in Scala 2.13 CI Sub-task Resolved Takeshi Yamamuro
        85.
        Upgrade SBT to 1.5.2 Sub-task Resolved Dongjoon Hyun
        86.
        Sort elements generated by collect_set in SQLQueryTestSuite Sub-task Resolved Takeshi Yamamuro
        87.
        Upgrade scalatestplus artifacts to 3.2.9.0 Sub-task Resolved Kousuke Saruta
        88.
        Upgrade Scala 2.13 to 2.13.7 Sub-task In Progress Unassigned
        89.
        Re-cleanup `procedure syntax is deprecated` compilation warning in Scala 2.13 Sub-task Resolved Yang Jie
        90.
        Ensure mllib and kafka-0-10 module can maven test independently in Scala 2.13 Sub-task Resolved Yang Jie
        91.
        Turn compilation warnings related to `procedure syntax` to errors in Scala 2.13 Sub-task Resolved Yang Jie
        92.
        Upgrade SBT to 1.5.3 Sub-task Resolved Kousuke Saruta
        93.
        Upgrade SBT to 1.5.5 Sub-task Resolved Kousuke Saruta
        94.
        Upgrade scala-maven-plugin to 4.5.3 Sub-task Resolved William Hyun
        95.
        Upgrade sbt-mima-plugin to 0.9.2 Sub-task Resolved Dongjoon Hyun
        96.
        Ensure all modules can be maven test independently in Scala 2.13 Sub-task Resolved Yang Jie
        97.
        Disable MiMa for Scala 2.13 artifacts Sub-task Resolved Dongjoon Hyun
        98.
        Support Scala 2.13 test in `dev/run-tests.py` Sub-task Resolved Dongjoon Hyun
        99.
        Upgrade GenJavadoc to 0.18 Sub-task Resolved Kousuke Saruta
        100.
        Add Scala 2.13 daily build and test GitHub Action job Sub-task Resolved Dongjoon Hyun

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              MasseGuillaume Guillaume Massé
              Shepherd:
              Sean R. Owen
            • Votes:
              16 Vote for this issue
              Watchers:
              56 Start watching this issue

              Dates

              • Created:
                Updated: