Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25075

Build and test Spark against Scala 2.13

    XMLWordPrintableJSON

Details

    Description

      This umbrella JIRA tracks the requirements for building and testing Spark against the current Scala 2.13 milestone.

      Attachments

        1.
        Use scala-xml explicitly Sub-task Resolved Unassigned
        2.
        Remove usage of Traversable Sub-task Resolved Sean R. Owen
        3.
        Avoid use of Scala collection classes that are removed in 2.13 Sub-task Resolved Sean R. Owen
        4.
        Remove usage of TraversableOnce Sub-task Resolved Sean R. Owen
        5.
        Upgrade breeze to 1.0 Sub-task Resolved Sean R. Owen
        6.
        Update scalatest, scalacheck, scopt, scala-parser-combinators, clapper classutil Sub-task Resolved Sean R. Owen
        7.
        Update twitter chill for 2.13 Sub-task Resolved Sean R. Owen
        8.
        Remove use of procedure-like method definition for 2.13 Sub-task Resolved Sean R. Owen
        9.
        Fix internal usages of mutable collection for Seq in 2.13 Sub-task Resolved Sean R. Owen
        10.
        Move scalafmt to Scala 2.12 profile; bump to 0.12 Sub-task Resolved Sean R. Owen
        11.
        Update Kafka to 2.4.0 for Scala 2.13 Sub-task Resolved Jungtaek Lim
        12.
        Use scala-parallel-collections library in 2.13 Sub-task Resolved Sean R. Owen
        13.
        Update REPL for 2.13 Sub-task Resolved Karol Chmist
        14.
        Remove use of deprecated symbol literal " 'name " syntax in favor Symbol("name") Sub-task Resolved Sean R. Owen
        15.
        Replace ambiguous varargs call parallelize(Array) with parallelize(Seq) Sub-task Resolved Sean R. Owen
        16.
        Remove use of Unit companion object in favor of () Sub-task Resolved Sean R. Owen
        17.
        Remove use of TraversableLike Sub-task Resolved Sean R. Owen
        18.
        Replace Array.deep comparisons with Array.sameElements Sub-task Resolved Sean R. Owen
        19.
        Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 Sub-task Resolved Dongjoon Hyun
        20.
        Replace Ordering.Double with Ordering.Double.TotalOrdering Sub-task Resolved Sean R. Owen
        21.
        Remove deprecated SparkConf.setAll(Traversable) Sub-task Resolved Sean R. Owen
        22.
        Use DoubleAsIfIntegral / FloatAsIfIntegral workalike in 2.13 Sub-task Resolved Sean R. Owen
        23.
        Change classes extending scala collection classes to work with 2.13 Sub-task Resolved Sean R. Owen
        24.
        No parens for calls to BigDecimal.longValue() etc. Sub-task Resolved Sean R. Owen
        25.
        Scala 2.13 compile errors from Hadoop LocalFileSystem subclasses Sub-task Closed Unassigned
        26.
        Resolve ambiguous overload from RegexParsers in 2.13 Sub-task Resolved Sean R. Owen
        27.
        Resolve Array + reference type compile problems in 2.13, with sc.parallelize Sub-task Resolved Sean R. Owen
        28.
        Some imports, function need more explicit resolution in 2.13 Sub-task Resolved Sean R. Owen
        29.
        Upgrade Genjavadoc to 0.15 Sub-task Resolved Kousuke Saruta
        30.
        Iterator doesn't refer outer identifier named "iterator" properly in Scala 2.13 Sub-task Resolved Kousuke Saruta
        31.
        Get tests working for Scala 2.13 build Sub-task Resolved Sean R. Owen
        32.
        Upgrade to scalatest 3.2.0 for Scala 2.13.3 compatibility Sub-task Resolved Sean R. Owen
        33.
        Upgrade Genjavadoc to 0.16 Sub-task Resolved Dongjoon Hyun
        34.
        Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts Sub-task Resolved Dongjoon Hyun
        35.
        Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal Sub-task Resolved Dongjoon Hyun
        36.
        Use HashMap.withDefaultValue in RDDSuite Sub-task Resolved Dongjoon Hyun
        37.
        Make BlockManagerSuite robust from Scala object size difference Sub-task Resolved Dongjoon Hyun
        38.
        Update json4s for Scala 2.13 Sub-task Resolved Dongjoon Hyun
        39.
        Fix TaskSetManagerSuite by hiding `org.apache.spark.FakeSchedulerBackend` Sub-task Resolved Dongjoon Hyun
        40.
        Fix testCommandAvailable to use POSIX compatible `command -v` Sub-task Resolved Hyukjin Kwon
        41.
        Fix HistoryServerSuite in Scala 2.13 Sub-task Resolved Unassigned
        42.
        Use === instead IndexSeqView Sub-task Resolved Dongjoon Hyun
        43.
        ResourceAllocator.availableAddrs should be deterministic Sub-task Resolved Dongjoon Hyun
        44.
        JsonProtocol.accumulablesToJson should be deterministic Sub-task Resolved Dongjoon Hyun
        45.
        Remove javax.ws.rs.NotFoundException from `import` in StagesResource/OneApplicationResource Sub-task Resolved Dongjoon Hyun
        46.
        Pass all `core` module UTs in Scala 2.13 Sub-task Resolved Dongjoon Hyun
        47.
        Let sql/catalyst module tests pass for Scala 2.13 Sub-task Resolved Yang Jie
        48.
        Fix CaseInsensitiveMap in Scala 2.13 Sub-task Resolved Karol Chmist
        49.
        Pass all `sql/core` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        50.
        Ignore deprecation warnings for build with Scala-2.13 and sbt Sub-task Resolved Kousuke Saruta
        51.
        Append toMap to Map#filterKeys if the result of filter is concatenated with another Map for Scala 2.13 Sub-task Resolved Kousuke Saruta
        52.
        Fix code which causes error when build with sbt and Scala 2.13 Sub-task Resolved Kousuke Saruta
        53.
        Pass all `sql/hive` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        54.
        Pass all `sql/hive-thriftserver` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        55.
        Add Scala 2.13 build test in GitHub Action Sub-task Resolved Dongjoon Hyun
        56.
        Pass all `external/avro` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        57.
        Pass all `streaming` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        58.
        Pass all `mllib` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        59.
        Pass all `mesos` module UTs in Scala 2.13 Sub-task Resolved Yang Jie
        60.
        Add a Jenkins build and test job for Scala 2.13 Sub-task Resolved Shane Knapp
        61.
        Update how to build doc for Scala 2.13 with sbt Sub-task Resolved Kousuke Saruta
        62.
        Fix SparkBuild.scala to recognize build settings for Scala 2.13 Sub-task Resolved Kousuke Saruta
        63.
        Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job Sub-task Resolved Kousuke Saruta
        64.
        Rename classOf[Seq] to classOf[scala.collection.Seq] in PostgresIntegrationSuite for Scala 2.13 Sub-task Resolved Kousuke Saruta
        65.
        Fix procedure-like declaration compilation warning in Scala 2.13 Sub-task Resolved Yang Jie
        66.
        Upgrade to Scala 2.13.4 Sub-task Resolved Dongjoon Hyun
        67.
        Upgrade Scala 2.13 from 2.13.3 to 2.13.4 Sub-task Closed Unassigned
        68.
        Change `InMemoryTable` not to use Tuple.hashCode for `BucketTransform` Sub-task Resolved Dongjoon Hyun
        69.
        Suppress unimportant compilation warnings in Scala 2.13 Sub-task Resolved Yang Jie
        70.
        Word2VecSuite failed for Scala 2.13 Sub-task Resolved koert kuipers
        71.
        ExpressionEncoderSuite failed in spark-branch-3.1-test-maven-hadoop-*-jdk-*-scala-2.13 Sub-task Resolved Yang Jie
        72.
        HadoopVersionInfoSuite faild when maven test in Scala 2.13 Sub-task Closed Unassigned
        73.
        Java UT failed when testing sql/hive module independently in Scala 2.13 Sub-task Resolved Yang Jie
        74.
        Add Scala 2.13 packaging and publishing Sub-task Resolved Dongjoon Hyun
        75.
        Upgrade Scala to avoid Scala's bug about illegal cyclic reference Sub-task Closed Unassigned
        76.
        Upgrade Scala to 2.13.5 Sub-task Resolved Dongjoon Hyun
        77.
        Update sbt version to 1.4.9 Sub-task Resolved William Hyun
        78.
        Spark artefacts built against Scala 2.13 incorrectly depend on Scala 2.12 Sub-task Resolved Sean R. Owen
        79.
        Jekyll fails to generate Scala API docs for Scala 2.13 Sub-task Resolved Kousuke Saruta
        80.
        Upgrade SBT to 1.5.0 Sub-task Resolved Dongjoon Hyun
        81.
        Suppress `symbol literal is deprecated` compilation warnings in Scala 2.13 Sub-task Resolved Unassigned
        82.
        Upgrade GenJavadoc to 0.17 Sub-task Resolved Kousuke Saruta
        83.
        Many test cases failed in Scala 2.13 CI Sub-task Resolved Takeshi Yamamuro
        84.
        Upgrade SBT to 1.5.2 Sub-task Resolved Dongjoon Hyun
        85.
        Sort elements generated by collect_set in SQLQueryTestSuite Sub-task Resolved Takeshi Yamamuro
        86.
        Upgrade scalatestplus artifacts to 3.2.9.0 Sub-task Resolved Kousuke Saruta
        87.
        Re-cleanup `procedure syntax is deprecated` compilation warning in Scala 2.13 Sub-task Resolved Yang Jie
        88.
        Ensure mllib and kafka-0-10 module can maven test independently in Scala 2.13 Sub-task Resolved Yang Jie
        89.
        Turn compilation warnings related to `procedure syntax` to errors in Scala 2.13 Sub-task Resolved Yang Jie
        90.
        Upgrade SBT to 1.5.3 Sub-task Resolved Kousuke Saruta
        91.
        Upgrade SBT to 1.5.5 Sub-task Resolved Kousuke Saruta
        92.
        Upgrade scala-maven-plugin to 4.5.3 Sub-task Resolved William Hyun
        93.
        Upgrade sbt-mima-plugin to 0.9.2 Sub-task Resolved Dongjoon Hyun
        94.
        Ensure all modules can be maven test independently in Scala 2.13 Sub-task Resolved Yang Jie
        95.
        Disable MiMa for Scala 2.13 artifacts Sub-task Resolved Dongjoon Hyun
        96.
        Support Scala 2.13 test in `dev/run-tests.py` Sub-task Resolved Dongjoon Hyun
        97.
        Upgrade GenJavadoc to 0.18 Sub-task Resolved Kousuke Saruta
        98.
        Add Scala 2.13 daily build and test GitHub Action job Sub-task Resolved Dongjoon Hyun
        99.
        SparkContextSuite random failure in Scala 2.13 Sub-task Resolved Yang Jie
        100.
        Published 2.13 POM lists `scala-parallel-collections` only in `scala-2.13` profile Sub-task Resolved Lukas Rytz

        Activity

          People

            srowen Sean R. Owen
            MasseGuillaume Guillaume Massé
            Sean R. Owen Sean R. Owen
            Votes:
            16 Vote for this issue
            Watchers:
            60 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: