[SPARK-44111] Prepare Apache Spark 4.0.0

• Type: Umbrella
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Build
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Dongjoon Hyun
• Created: 20/Jun/23 17:45
• Updated: 15/Nov/24 18:46

Description

For now, this issue aims to collect ideas for planning Apache Spark 4.0.0.

We will add more items which will be excluded from Apache Spark 3.5.0 (Feature Freeze: July 16th, 2023).

Spark 1: 2014.05 (1.0.0) ~ 2016.11 (1.6.3)
Spark 2: 2016.07 (2.0.0) ~ 2021.05 (2.4.8)
Spark 3: 2020.06 (3.0.0) ~ 2026.xx (3.5.x)
Spark 4: 2025.02 (4.0.0, NEW)
- January 15th 2025	Code freeze. Release branch cut.
- February 1st 2025	QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.
- February 15th 2025	Release candidates (RC), voting, etc. until final release passes

Issue Links

• relates to: SPARK-44112 Drop Java 8 and 11 support
• relates to: SPARK-48918 Create a unified SQL Scala interface shared by regular SQL and Connect.
• relates to: SPARK-48338 Sql Scripting support for Spark SQL
• relates to: SPARK-45869 Revisit and Improve Spark Standalone Cluster
• relates to: SPARK-44937 Add SSL/TLS support for RPC and Shuffle communications
• relates to: SPARK-46837 String function support (parent)
• relates to: SPARK-43351 Support Golang in Spark Connect
• relates to: SPARK-43831 Build and Run Spark on Java 21
• relates to: SPARK-45923 Spark Kubernetes Operator
• relates to: SPARK-44124 Upgrade AWS SDK to v2
• relates to: SPARK-35801 SPIP: Row-level operations in Data Source V2
• relates to: SPARK-42551 Support more subexpression elimination cases
• relates to: SPARK-44183 Increase PyArrow minimum version to 4.0.0
• relates to: SPARK-47361 Improve JDBC data sources
• relates to: SPARK-43836 Make Scala 2.13 as default Scala version in Spark 3.5
• relates to: SPARK-47970 Revisit skipped parity tests for PySpark Connect
• relates to: SPARK-49555 SQL Pipe Syntax
• relates to: SPARK-37935 Migrate onto error classes
• relates to: SPARK-48094 Reduce GitHub Action usage according to ASF project allowance
• relates to: SPARK-45314 Drop Scala 2.12 and make Scala 2.13 by default
• relates to: SPARK-45315 Drop JDK 8/11 and make JDK 17 by default
• relates to: SPARK-47540 SPIP: Pure Python Package (Spark Connect)
• relates to: SPARK-44101 Support pandas 2
• relates to: SPARK-44728 Improve PySpark documentations
• relates to: SPARK-44893 ThreadInfo improvements for monitoring APIs
• relates to: SPARK-45981 Improve Python language test coverage
• relates to: SPARK-47046 Apache Spark 4.0.0 Dependency Audit and Cleanup
• relates to: SPARK-49524 Improve K8s support
• relates to: SPARK-50266 Improve Codegen Support
• links to: Apache Spark 4.0.0 Release Window

Sub-Tasks

• Drop mesos support: Resolved
• Drop Python 3.8 support: Resolved
• Deprecate SaprkR: Resolved
• Remove deprecated `BinaryClassificationMetrics.scoreLabelsWeight`: Resolved
• Fix `GenerateMIMAIgnore.isPackagePrivateModule` to return false instead of failing: Resolved
• Remove shim classes for Hive prior 2.0.0: Resolved
• Support Hive 4.0 metastore: Resolved
• Cast between UDT and other types: Resolved
• Eliminate unnecessary reflection invocation in Hive shim classes: Resolved
• Deprecated no-op `spark.shuffle.spill` config: Resolved
• Deprecate spark.sql.parser.escapedStringLiterals: Resolved
• Deprecate `spark.sql.legacy.allowZeroIndexInFormatString`: Resolved
• Deprecate spark.network.remoteReadNioBufferConversion: Resolved
• Upgrade Guava to 33.2.1-jre: Resolved
• Upgrade Guava to 33.3.1-jre: Resolved
• Upgrade Scala to 2.13.12: Resolved
• Upgrade Scala to 2.13.13: Resolved
• Upgrade Scala to 2.13.14: Resolved
• Upgrade Scala to 2.13.15: Resolved
• Enable spark.shuffle.service.removeShuffle by default: Resolved
• Enable spark.eventLog.compress by default: Resolved
• Enable spark.eventLog.rolling.enabled by default: Resolved
• Enable `spark.metrics.appStatusSource.enabled` by default: Resolved
• Enable `spark.stage.ignoreDecommissionFetchFailure` by default: Resolved
• Enable `spark.sql.sources.v2.bucketing.pushPartValues.enabled` by default: Resolved
• Update `spark.speculation.multiplier` to 3 and `spark.speculation.quantile` to 0.9: Resolved
• Change default of spark.sql.legacy.timeParserPolicy from EXCEPTION to CORRECTED: Resolved
• Make EventLoggingListenerSuite independent from spark.eventLog.compress conf: Resolved
• Fix EventLogFileWriters to handle `none` codec case: Resolved
• Migrate antlr4 from 4.9 to 4.10+: Resolved
• Upgrade Python to 3.11 in Maven builds: Resolved
• Upgrade `protobuf` to 4.25.1 to support `Python 3.11`: Resolved
• Upgrade `protobuf-java` to 3.25.1 to match with protobuf 4.25.1: Resolved
• Upgrade `protobuf-java` to 4.28.3: Resolved
• Upgrade protobuf-java to 4.29.1: Resolved
• Support Python 3.12: Resolved
• Support Python 3.13 in Spark Classic: Resolved
• Remove pinned version of torch for Python 3.12 support: Resolved
• Handle PEP-632 by checking `ModuleNotFoundError` on `setuptools` in Python 3.12: Resolved
• `mypy` should have `--python-executable` parameter: Resolved
• Upgrade Pandas to 2.2.0: Resolved
• Update pandas to 2.2.1: Resolved
• Update pandas to 2.2.2: Resolved
• Update pandas to 2.2.3: Resolved
• Remove `distutils` usage: Resolved
• Remove deprecated Hadoop-2 `LocatedFileStatus` constructor: Resolved
• Improve InMemoryFileIndex to use FileSystem.listFiles API: Resolved
• Change RocksDB as default shuffle service db backend: Resolved
• Remove threeten-extra exclusion in enforceBytecodeVersion rule: Resolved
• Upgrade ORC to 1.9.1: Resolved
• Upgrade ORC to 1.9.2: Resolved
• Upgrade ORC to 2.0.0: Resolved
• Upgrade ORC to 2.0.1: Resolved
• Upgrade ORC to 2.0.2: Resolved
• Remove workaround for YARN-3350: Resolved
• Update docs for size function and sizeOfNull configuration: Resolved
• Support ORC Brotli codec: Resolved
• Fix ORC tests to be independent from default compression: Resolved
• Use `zstd` as the default ORC compression: Resolved
• Use the default ORC compression in OrcReadBenchmark: Resolved
• Improve `TPCDSQueryBenchmark` to support other file formats: Resolved
• Use default ORC compression in data source benchmarks: Resolved
• Upgrade `Parquet` to 1.14.1: Resolved
• Upgrade Parquet to 1.14.2: Resolved
• Upgrade `Parquet` to 1.14.3: Resolved
• Bump Apache Parquet to 1.14.4: Resolved
• Bump Apache Parquet to 1.15.0: Resolved
• Upgrade Avro to 1.11.3: Resolved
• Upgrade Avro to 1.12.0: Resolved
• Documented Prometheus endpoints: Resolved
• Document a few missed `spark.ui.*` configs to `Configuration` page: Resolved
• Upgrade Maven to 3.9.6 for MNG-7913: Resolved
• Upgrade Maven to 3.9.7: Resolved
• Upgrade Maven to 3.9.8: Resolved
• Upgrade maven to 3.9.9: Resolved
• Use Scala 2.13 Spark distribution in HiveExternalCatalogVersionsSuite: Resolved
• Add Apple Silicon Maven build test to GitHub Action CI: Resolved
• Add Daily Apple Silicon Github Action Job (Java/Scala): Resolved
• Migrate from AppVeyor to GitHub Actions for SparkR tests on Windows: Resolved
• Attach codec extension to avro datasource files: Resolved
• Benchmarking Avro with Compression Codecs: Resolved
• Codec xz and zstandard support compression level for avro files: Resolved
• Disable unsupported `ExtendedLevelDBTest` on `MacOS/aarch64`: Resolved
• Change to use bcprov/bcpkix-jdk18on for test: Resolved
• Add `bouncy-castle` test dependencies to `sql/core` module for Hadoop 3.4.0: Resolved
• Add `bcpkix-jdk18on` test dependencies to `hive` module for Hadoop 3.4.0: Resolved
• Upgrade `bouncycastle` to 1.78: Resolved
• Use Hadoop 3.3.5 winutils in AppVeyor build: Resolved
• Upgrade Hadoop to 3.3.6: Resolved
• Upgrade Hadoop to 3.4.0: Resolved
• Upgrade Hadoop to 3.4.1: Resolved
• Use hadoop 3.4.0 in some docs: Resolved
• Fix `IsolatedClientLoader.supportsHadoopShadedClient` to handle Hadoop 3.4+: Resolved
• Exclude `logback` dependency from SBT like Maven: Resolved
• Ignore `IntentionallyFaultyConnectionProvider` error in `CliSuite`: Resolved
• Set spark.hadoop.fs.s3a.connection.establish.timeout to 30s: Resolved
• Regenerate benchmark results: Resolved
• Upgrade R version from 4.3.1 to 4.3.2 in AppVeyor: Resolved
• Use R 4.3.3 in `windows` R GitHub Action job: Resolved
• Use `Ubuntu 22.04` in `dev/infra/Dockerfile`: Resolved
• Support MergeInto in DataFrameWriterV2: Resolved
• Upgrade Arrow to 14.0.0: Resolved
• Upgrade pyarrow to 14: Resolved
• Upgrade Arrow to 14.0.1: Resolved
• Upgrade Arrow to 14.0.2: Resolved
• Upgrade Arrow to 15.0.0: Resolved
• Upgrade Arrow to 15.0.2: Resolved
• Upgrade pyarrow to 15.0.0: Resolved
• Upgrade `Arrow` to 16.0.0: Resolved
• Upgrade `Arrow` to 17.0.0: Resolved
• fail by default in deprecated catalog plugin APIs: Resolved
• Refresh testing image for pyarrow 17: Resolved
• Upgrade the minimum version of PyArrow to 10.0.0: Resolved
• Upgrade the minimum version of `arrow` R package to 10.0.0: Resolved
• Move `o.a.s.variant` to `o.a.s.types.variant`: Resolved
• Remove Spark 3.0~3.2 pyspark/version.py workaround from release scripts: Resolved
• Add `slf4j-api` jar to the class path first before the others of `jars` directory: Resolved
• Make Spark build with -release instead of -target: Resolved
• Use `HiveConf.getConfVars` or Hive conf names directly: Resolved
• Upgrade hive-service-rpc 4.0.0: Resolved
• Upgrade Kafka to 3.6.1: Resolved
• Upgrade Kafka to 3.7.0: Resolved
• Upgrade Kafka to 3.7.1: Resolved
• Upgrade Kafka to 3.8.0: Resolved
• Upgrade Kafka to 3.8.1: Resolved
• Upgrade Kafka to 3.9.0: Resolved
• Fix `dev/scalastyle` to check `hadoop-cloud` and `jvm-profiler` modules: Resolved
• Remove redundant rules from `MimaExcludes`: Resolved
• Run `ANSI` SQL CI twice per day: Resolved
• Use ANSI SQL mode by default: Resolved
• Switch ANSI SQL CI job to NON-ANSI SQL CI job: Resolved
• Update `sql-ref-ansi-compliance.md` and `sql-ref-identifier.md`: Resolved
• Fix a bug in try_divide function when with decimals: Resolved
• Regenerate benchmark results after turning ANSI on: Resolved
• Remove install_scala from build/mvn: Resolved
• Use Hive tables explicitly for Hive table capability tests: Resolved
• Parameterize max limits of `spark.sql.test.randomDataGenerator`: Resolved
• Set `spark.sql.legacy.createHiveTableByDefault` to `false` by default: Resolved
• Support SPARK_SQL_LEGACY_CREATE_HIVE_TABLE env variable: Resolved
• Fix StreamingQueryHashPartitionVerifySuite to update golden files correctly: Resolved
• Introduces a universal BinaryFormatter to make binary output consistent: Resolved
• Disable a flaky `SparkSessionE2ESuite.interrupt tag` test: Resolved
• Add toArrow() DataFrame method to PySpark: Resolved
• Mark deprecated functions with `@deprecated` in `SQLImplicits`: Resolved
• Allow passing PyArrow Table to createDataFrame(): Resolved
• Support additional PyArrow Table column types: Resolved
• Fix `Black` target version to Python 3.9: Resolved
• Make `IndexShuffleBlockResolver` have explicit constructors: Resolved
• Preserve nulls in map columns in PyArrow Tables: Resolved
• Use R 4.4.0 in `windows` R GitHub Action Windows job: Resolved
• Use R 4.4.1 in `windows` R GitHub Action Windows job: Resolved
• Use R 4.4.2 in `windows` R GitHub Action Windows job: Resolved
• Replace SQLContext with SparkSession in STS: Resolved
• Implement IndentingXMLStreamWriter: Resolved
• `Row#getSeq` exhibits a performance regression between master and Spark 3.5 with Scala 2.12: Resolved
• Regenerate benchmark results: Resolved
• Investigate ExternalAppendOnlyUnsafeRowArrayBenchmark: Resolved
• Add `scalastyle` and `checkstyle` rules to avoid `URL` constructors: Resolved
• Enable ANSI for TPCDSQueryBenchmark: Resolved
• Support CRC32C for Shuffle Checksum: Resolved
• Free memory allocated by large vectors when vectors are reset: Resolved
• Remove `Python 3.6 and older`-related logic from `try_simplify_traceback`: Resolved
• Fix `assembly/pom.xml` to use `{project.version}` instead of `{version}`: Resolved
• Use more appropriate parameter type to construct `GenericArrayData`: Resolved
• Support user-defined labels for OnDemand PVCs: Resolved
• Fix `spark-rm` and `infra` docker files to create `pypy3.9` links: Resolved
• Remove the flaky `EXEC IMMEDIATE STACK OVERFLOW` test case: Resolved
• Re-enable test_create_dataframe_from_pandas_with_day_time_interval: Resolved
• Pick up system property defaults in SharedSparkContext test suite mixin: Resolved
• Make `docs/index.md` up-to-date for 4.0.0: Resolved
• Link `python3` to `python3.9` in `spark-rm` Docker image: Resolved
• Support `spark.test.master` in `SparkSubmitArguments`: Resolved
• Parallelize Sphinx build of Python API docs: Resolved
• Limit `Sphinx` build parallelism to 4 by default: Resolved
• Deflaky PythonForeachWriterSuite: Resolved
• Function `substring` should accept column names: Resolved
• Add official image Dockerfile for Apache Spark 4.0.0-preview1: Resolved
• Add Apache Spark `4.0.0-preview2` Dockerfiles: Resolved
• Increase `spark.test.docker.connectionTimeout` to 10min: Resolved
• Change log level to debug in BlockManagerMasterEndpoint: Resolved
• Set `OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES` in `MacOS` GitHub Action Job: Resolved
• Use `MacOS 15` in `build_maven_java21_macos14.yml`: Resolved
• Upgrade `Arrow` to 18.0.0: Resolved
• Skip PySpark `test_cast_to_udt_with_udt` test during `SPARK_SKIP_CONNECT_COMPAT_TESTS`: Resolved
• Fix `SPARK-41388` test case to assert the result: Resolved
• Use SPDX short identifier in `licenses` name: Resolved
• Use `lower_camel_case` in `store_types.proto`: Resolved
• Enable `buf-lint-action` on `core` module: Resolved
• Reduce Cloud Store List API cost for state store maintenance task: Resolved
• RocksDB State Store Doesn't Need to Recheck checkpoint path existence: Resolved
• Increase test timeout of PythonForeachWriterSuite to 60s: Resolved
• Deflaky `(Ssl)CoarseGrainedExecutorBackendSuite` to reduce test resource and increase timeout: Resolved
• Fix `AsyncRDDActionsSuite.FutureAction result, timeout` test case to be robust: Resolved
• Remove redundant `blank space` after 'show' in `Scala` and `Connect` clients: Resolved
• Make HadoopMapReduceCommitProtocol respect 'mapreduce.output.basename' to generate file names: Resolved
• Handle or document `NumPy 2.1` difference in Python 3.13: Resolved
• The `sbt console` command is not available: Resolved
• Upgrade the minimum version of `googleapis-common-protos` to 1.65.0: Resolved
• Upgrade `protobuf` Python package to 5.28.3: Resolved
• Fix `protobuf` module Maven compilation: Resolved
• Upgrade ORC to 2.0.3: Resolved
• Upgrade `grpcio*` to 1.59.3 for Python 3.12: Resolved
• Update INFRA docker image to use `jammy-20240911.1`: Resolved
• Flaky test: `SPARK-46957: Migrated shuffle files should be able to cleanup from executor` with Java 21: Resolved
• Remove `run-tests-jenkins`-related stuff: Resolved
• Flaky test: `Cancellation APIs in SparkSession are isolated`: Resolved
• Install Jinja2 for Python 3.13 build: Resolved
• Disable `SparkSessionJobTaggingAndCancellationSuite.Cancellation APIs in SparkSession are isolated`: Resolved
• Upgrade `bouncycastle` to 1.79: Resolved
• Add `getSystemProperty` to PySpark `SparkContext`: Resolved
• Make `curl` retry 3 times in `bin/mvn`: Resolved
• Make GA's `build_error_docs` run only once: Resolved
• Make `JdbcDialect.classifyException` throw out the original exception: Resolved
• Upgrade oracle jdbc driver to 23.6.0.24.10: Resolved
• Support Table Options for Insert SQL: Resolved
• Use mirror host instead of `archive.apache.org`: Resolved
• Remove `(any|empty).proto` from RAT exclusion: Resolved
• Support Python 3.13 in Spark Connect: Resolved
• Investigate UTF8_BINARY regression: Resolved
• Make `--remote` an official option by removing `experimental` warning: Resolved
• Enable ANSI in SQL *SQLQueryTestSuite by default: Resolved
• Rename `ojdbc11.version` to `ojdbc17.version`: Resolved
• Make `Parameterized SQL queries` of `SparkSession.sql` API GA: Resolved
• Remove workaround for THRIFT-4805: Resolved
• Proper error handling for unsupported file system when reading files: Resolved
• Update cross-spawn JS module to surpress a warning in lint: Resolved
• Fix `spark.storage.replication.proactive` default value documentation: Resolved
• migrate kafka consumer offset information in spark to new MSK cluster: Open
• Re-enable `SparkSessionE2ESuite.interrupt tag`: Reopened
• Reenble AmmoniteTest tests in Maven builds: Open
• Re-enable `SparkSessionJobTaggingAndCancellationSuite.Cancellation APIs in SparkSession are isolated`: Open
• Add addArtifact API to the Spark SQL Core: Resolved
• Add addArtifact API to PySpark: Open
• Log downloaded archive file sizes in SparkContext and Executor: Resolved
• Use `rouge` `4.5.x` by remove the upper bound `3.x`: Resolved
• Add quote builtin function: Open
• Use CommonJS format for ESLint configuration file: Open