[SPARK-25075] Build and test Spark against Scala 2.13

• Type: Umbrella
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: Build, MLlib, Project Infra, Spark Core, SQL
• Target Version/s: 3.2.0
• Assignee: Unassigned
• Reporter: Guillaume Massé
• Shepherd: Sean R. Owen
• Created: 09/Aug/18 13:26
• Updated: 23/Feb/21 14:20

Description

This umbrella JIRA tracks the requirements for building and testing Spark against the current Scala 2.13 milestone.

Sub-Tasks

• Use scala-xml explicitly: Resolved
• Remove usage of Traversable: Resolved
• Avoid use of Scala collection classes that are removed in 2.13: Resolved
• Remove usage of TraversableOnce: Resolved
• Upgrade breeze to 1.0: Resolved
• Update scalatest, scalacheck, scopt, scala-parser-combinators, clapper classutil: Resolved
• Update twitter chill for 2.13: Resolved
• Remove use of procedure-like method definition for 2.13: Resolved
• Fix internal usages of mutable collection for Seq in 2.13: Resolved
• Move scalafmt to Scala 2.12 profile; bump to 0.12: Resolved
• Update Kafka to 2.4.0 for Scala 2.13: Resolved
• Use scala-parallel-collections library in 2.13: Resolved
• Update REPL for 2.13: Resolved
• Remove use of deprecated symbol literal " 'name " syntax in favor Symbol("name"): Resolved
• Replace ambiguous varargs call parallelize(Array) with parallelize(Seq): Resolved
• Remove use of Unit companion object in favor of (): Resolved
• Remove use of TraversableLike: Resolved
• Replace Array.deep comparisons with Array.sameElements: Resolved
• Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1: Resolved
• Replace Ordering.Double with Ordering.Double.TotalOrdering: Resolved
• Remove deprecated SparkConf.setAll(Traversable): Resolved
• Use DoubleAsIfIntegral / FloatAsIfIntegral workalike in 2.13: Resolved
• Change classes extending scala collection classes to work with 2.13: Resolved
• No parens for calls to BigDecimal.longValue() etc.: Resolved
• Scala 2.13 compile errors from Hadoop LocalFileSystem subclasses: Closed
• Resolve ambiguous overload from RegexParsers in 2.13: Resolved
• Resolve Array + reference type compile problems in 2.13, with sc.parallelize: Resolved
• Some imports, function need more explicit resolution in 2.13: Resolved
• Upgrade Genjavadoc to 0.15: Resolved
• Iterator doesn't refer outer identifier named "iterator" properly in Scala 2.13: Resolved
• Get tests working for Scala 2.13 build: Resolved
• Upgrade to scalatest 3.2.0 for Scala 2.13.3 compatibility: Resolved
• Upgrade Genjavadoc to 0.16: Resolved
• Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts: Resolved
• Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal: Resolved
• Use HashMap.withDefaultValue in RDDSuite: Resolved
• Make BlockManagerSuite robust from Scala object size difference: Resolved
• Update json4s for Scala 2.13: Resolved
• Fix TaskSetManagerSuite by hiding `org.apache.spark.FakeSchedulerBackend`: Resolved
• Fix testCommandAvailable to use POSIX compatible `command -v`: Resolved
• Fix HistoryServerSuite in Scala 2.13: Resolved
• Use === instead IndexSeqView: Resolved
• ResourceAllocator.availableAddrs should be deterministic: Resolved
• JsonProtocol.accumulablesToJson should be deterministic: Resolved
• Remove javax.ws.rs.NotFoundException from `import` in StagesResource/OneApplicationResource: Resolved
• Pass all `core` module UTs in Scala 2.13: Resolved
• Let sql/catalyst module tests pass for Scala 2.13: Resolved
• Fix CaseInsensitiveMap in Scala 2.13: Resolved
• Pass all `sql/core` module UTs in Scala 2.13: Resolved
• Ignore deprecation warnings for build with Scala-2.13 and sbt: Resolved
• Append toMap to Map#filterKeys if the result of filter is concatenated with another Map for Scala 2.13: Resolved
• Fix code which causes error when build with sbt and Scala 2.13: Resolved
• Pass all `sql/hive` module UTs in Scala 2.13: Resolved
• Pass all `sql/hive-thriftserver` module UTs in Scala 2.13: Resolved
• Add Scala 2.13 build test in GitHub Action: Resolved
• Pass all `external/avro` module UTs in Scala 2.13: Resolved
• Pass all `streaming` module UTs in Scala 2.13: Resolved
• Pass all `mllib` module UTs in Scala 2.13: Resolved
• Pass all `mesos` module UTs in Scala 2.13: Resolved
• Add a Jenkins build and test job for Scala 2.13: Resolved
• Update how to build doc for Scala 2.13 with sbt: Resolved
• Fix SparkBuild.scala to recognize build settings for Scala 2.13: Resolved
• Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job: Resolved
• Rename classOf[Seq] to classOf[scala.collection.Seq] in PostgresIntegrationSuite for Scala 2.13: Resolved
• Fix procedure-like declaration compilation warning in Scala 2.13: Resolved
• Upgrade to Scala 2.13.4: Resolved
• Upgrade Scala 2.13 from 2.13.3 to 2.13.4: Closed
• Change `InMemoryTable` not to use Tuple.hashCode for `BucketTransform`: Resolved
• Suppress unimportant compilation warnings in Scala 2.13: Resolved
• Word2VecSuite failed for Scala 2.13: Resolved
• ExpressionEncoderSuite failed in spark-branch-3.1-test-maven-hadoop-*-jdk-*-scala-2.13: Resolved
• HadoopVersionInfoSuite faild when maven test in Scala 2.13: Closed
• Java UT failed when testing sql/hive module independently in Scala 2.13: Resolved
• Add Scala 2.13 packaging and publishing: Resolved
• Upgrade Scala to avoid Scala's bug about illegal cyclic reference: Closed
• Replace symbol literals with Symbol constructor invocations to comply with Scala 2.13: In Progress
• Upgrade Scala to 2.13.5: Resolved
• Spark artefacts built against Scala 2.13 incorrectly depend on Scala 2.12: Resolved
• Jekyll fails to generate Scala API docs for Scala 2.13: Resolved