[SPARK-25075] Build and test Spark against Scala 2.13 - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Umbrella
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.0.0
Fix Version/s: 3.2.0
Component/s: Build, MLlib, Project Infra, Spark Core, SQL
Labels:
None

Target Version/s:

3.2.0

Description

This umbrella JIRA tracks the requirements for building and testing Spark against the current Scala 2.13 milestone.

Attachments

Sub-Tasks

1.	Use scala-xml explicitly	Resolved	Unassigned
2.	Remove usage of Traversable	Resolved	Sean R. Owen
3.	Avoid use of Scala collection classes that are removed in 2.13	Resolved	Sean R. Owen
4.	Remove usage of TraversableOnce	Resolved	Sean R. Owen
5.	Upgrade breeze to 1.0	Resolved	Sean R. Owen
6.	Update scalatest, scalacheck, scopt, scala-parser-combinators, clapper classutil	Resolved	Sean R. Owen
7.	Update twitter chill for 2.13	Resolved	Sean R. Owen
8.	Remove use of procedure-like method definition for 2.13	Resolved	Sean R. Owen
9.	Fix internal usages of mutable collection for Seq in 2.13	Resolved	Sean R. Owen
10.	Move scalafmt to Scala 2.12 profile; bump to 0.12	Resolved	Sean R. Owen
11.	Update Kafka to 2.4.0 for Scala 2.13	Resolved	Jungtaek Lim
12.	Use scala-parallel-collections library in 2.13	Resolved	Sean R. Owen
13.	Update REPL for 2.13	Resolved	Karol Chmist
14.	Remove use of deprecated symbol literal " 'name " syntax in favor Symbol("name")	Resolved	Sean R. Owen
15.	Replace ambiguous varargs call parallelize(Array) with parallelize(Seq)	Resolved	Sean R. Owen
16.	Remove use of Unit companion object in favor of ()	Resolved	Sean R. Owen
17.	Remove use of TraversableLike	Resolved	Sean R. Owen
18.	Replace Array.deep comparisons with Array.sameElements	Resolved	Sean R. Owen
19.	Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1	Resolved	Dongjoon Hyun
20.	Replace Ordering.Double with Ordering.Double.TotalOrdering	Resolved	Sean R. Owen
21.	Remove deprecated SparkConf.setAll(Traversable)	Resolved	Sean R. Owen
22.	Use DoubleAsIfIntegral / FloatAsIfIntegral workalike in 2.13	Resolved	Sean R. Owen
23.	Change classes extending scala collection classes to work with 2.13	Resolved	Sean R. Owen
24.	No parens for calls to BigDecimal.longValue() etc.	Resolved	Sean R. Owen
25.	Scala 2.13 compile errors from Hadoop LocalFileSystem subclasses	Closed	Unassigned
26.	Resolve ambiguous overload from RegexParsers in 2.13	Resolved	Sean R. Owen
27.	Resolve Array + reference type compile problems in 2.13, with sc.parallelize	Resolved	Sean R. Owen
28.	Some imports, function need more explicit resolution in 2.13	Resolved	Sean R. Owen
29.	Upgrade Genjavadoc to 0.15	Resolved	Kousuke Saruta
30.	Iterator doesn't refer outer identifier named "iterator" properly in Scala 2.13	Resolved	Kousuke Saruta
31.	Get tests working for Scala 2.13 build	Resolved	Sean R. Owen
32.	Upgrade to scalatest 3.2.0 for Scala 2.13.3 compatibility	Resolved	Sean R. Owen
33.	Upgrade Genjavadoc to 0.16	Resolved	Dongjoon Hyun
34.	Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts	Resolved	Dongjoon Hyun
35.	Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal	Resolved	Dongjoon Hyun
36.	Use HashMap.withDefaultValue in RDDSuite	Resolved	Dongjoon Hyun
37.	Make BlockManagerSuite robust from Scala object size difference	Resolved	Dongjoon Hyun
38.	Update json4s for Scala 2.13	Resolved	Dongjoon Hyun
39.	Fix TaskSetManagerSuite by hiding `org.apache.spark.FakeSchedulerBackend`	Resolved	Dongjoon Hyun
40.	Fix testCommandAvailable to use POSIX compatible `command -v`	Resolved	Hyukjin Kwon
41.	Fix HistoryServerSuite in Scala 2.13	Resolved	Unassigned
42.	Use === instead IndexSeqView	Resolved	Dongjoon Hyun
43.	ResourceAllocator.availableAddrs should be deterministic	Resolved	Dongjoon Hyun
44.	JsonProtocol.accumulablesToJson should be deterministic	Resolved	Dongjoon Hyun
45.	Remove javax.ws.rs.NotFoundException from `import` in StagesResource/OneApplicationResource	Resolved	Dongjoon Hyun
46.	Pass all `core` module UTs in Scala 2.13	Resolved	Dongjoon Hyun
47.	Let sql/catalyst module tests pass for Scala 2.13	Resolved	Yang Jie
48.	Fix CaseInsensitiveMap in Scala 2.13	Resolved	Karol Chmist
49.	Pass all `sql/core` module UTs in Scala 2.13	Resolved	Yang Jie
50.	Ignore deprecation warnings for build with Scala-2.13 and sbt	Resolved	Kousuke Saruta
51.	Append toMap to Map#filterKeys if the result of filter is concatenated with another Map for Scala 2.13	Resolved	Kousuke Saruta
52.	Fix code which causes error when build with sbt and Scala 2.13	Resolved	Kousuke Saruta
53.	Pass all `sql/hive` module UTs in Scala 2.13	Resolved	Yang Jie
54.	Pass all `sql/hive-thriftserver` module UTs in Scala 2.13	Resolved	Yang Jie
55.	Add Scala 2.13 build test in GitHub Action	Resolved	Dongjoon Hyun
56.	Pass all `external/avro` module UTs in Scala 2.13	Resolved	Yang Jie
57.	Pass all `streaming` module UTs in Scala 2.13	Resolved	Yang Jie
58.	Pass all `mllib` module UTs in Scala 2.13	Resolved	Yang Jie
59.	Pass all `mesos` module UTs in Scala 2.13	Resolved	Yang Jie
60.	Add a Jenkins build and test job for Scala 2.13	Resolved	Shane Knapp
61.	Update how to build doc for Scala 2.13 with sbt	Resolved	Kousuke Saruta
62.	Fix SparkBuild.scala to recognize build settings for Scala 2.13	Resolved	Kousuke Saruta
63.	Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job	Resolved	Kousuke Saruta
64.	Rename classOf[Seq] to classOf[scala.collection.Seq] in PostgresIntegrationSuite for Scala 2.13	Resolved	Kousuke Saruta
65.	Fix procedure-like declaration compilation warning in Scala 2.13	Resolved	Yang Jie
66.	Upgrade to Scala 2.13.4	Resolved	Dongjoon Hyun
67.	Upgrade Scala 2.13 from 2.13.3 to 2.13.4	Closed	Unassigned
68.	Change `InMemoryTable` not to use Tuple.hashCode for `BucketTransform`	Resolved	Dongjoon Hyun
69.	Suppress unimportant compilation warnings in Scala 2.13	Resolved	Yang Jie
70.	Word2VecSuite failed for Scala 2.13	Resolved	koert kuipers
71.	ExpressionEncoderSuite failed in spark-branch-3.1-test-maven-hadoop--jdk--scala-2.13	Resolved	Yang Jie
72.	HadoopVersionInfoSuite faild when maven test in Scala 2.13	Closed	Unassigned
73.	Java UT failed when testing sql/hive module independently in Scala 2.13	Resolved	Yang Jie
74.	Add Scala 2.13 packaging and publishing	Resolved	Dongjoon Hyun
75.	Upgrade Scala to avoid Scala's bug about illegal cyclic reference	Closed	Unassigned
76.	Upgrade Scala to 2.13.5	Resolved	Dongjoon Hyun
77.	Update sbt version to 1.4.9	Resolved	William Hyun
78.	Spark artefacts built against Scala 2.13 incorrectly depend on Scala 2.12	Resolved	Sean R. Owen
79.	Jekyll fails to generate Scala API docs for Scala 2.13	Resolved	Kousuke Saruta
80.	Upgrade SBT to 1.5.0	Resolved	Dongjoon Hyun
81.	Suppress `symbol literal is deprecated` compilation warnings in Scala 2.13	Resolved	Unassigned
82.	Upgrade GenJavadoc to 0.17	Resolved	Kousuke Saruta
83.	Many test cases failed in Scala 2.13 CI	Resolved	Takeshi Yamamuro
84.	Upgrade SBT to 1.5.2	Resolved	Dongjoon Hyun
85.	Sort elements generated by collect_set in SQLQueryTestSuite	Resolved	Takeshi Yamamuro
86.	Upgrade scalatestplus artifacts to 3.2.9.0	Resolved	Kousuke Saruta
87.	Re-cleanup `procedure syntax is deprecated` compilation warning in Scala 2.13	Resolved	Yang Jie
88.	Ensure mllib and kafka-0-10 module can maven test independently in Scala 2.13	Resolved	Yang Jie
89.	Turn compilation warnings related to `procedure syntax` to errors in Scala 2.13	Resolved	Yang Jie
90.	Upgrade SBT to 1.5.3	Resolved	Kousuke Saruta
91.	Upgrade SBT to 1.5.5	Resolved	Kousuke Saruta
92.	Upgrade scala-maven-plugin to 4.5.3	Resolved	William Hyun
93.	Upgrade sbt-mima-plugin to 0.9.2	Resolved	Dongjoon Hyun
94.	Ensure all modules can be maven test independently in Scala 2.13	Resolved	Yang Jie
95.	Disable MiMa for Scala 2.13 artifacts	Resolved	Dongjoon Hyun
96.	Support Scala 2.13 test in `dev/run-tests.py`	Resolved	Dongjoon Hyun
97.	Upgrade GenJavadoc to 0.18	Resolved	Kousuke Saruta
98.	Add Scala 2.13 daily build and test GitHub Action job	Resolved	Dongjoon Hyun
99.	SparkContextSuite random failure in Scala 2.13	Resolved	Yang Jie
100.	Published 2.13 POM lists `scala-parallel-collections` only in `scala-2.13` profile	Resolved	Lukas Rytz
101.	Upgrade Scala to 2.13.8	Resolved	Yang Jie

Activity

People

Assignee:: Sean R. Owen

Reporter:: Guillaume Massé

Shepherd:: Sean R. Owen

Votes:: 16 Vote for this issue

Watchers:: 60 Start watching this issue

Dates

Created:: 09/Aug/18 13:26

Updated:: 05/Mar/22 06:58

Resolved:: 13/Sep/21 16:08