Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33772

Build and Run Spark on Java 17

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • Build

    Description

      Apache Spark supports Java 8 and Java 11 (LTS). The next Java LTS version is 17.

      Version Release Date
      Java 17 (LTS) September 2021

      Apache Spark has a release plan and `Spark 3.2 Code freeze` was July along with the release branch cut.

      Supporting new Java version is considered as a new feature which we cannot allow to backport.

      Attachments

        Issue Links

          1.
          Add Java 17 ea build test to GitHub action Sub-task Resolved William Hyun  
          2.
          Update commons-lang3 to 3.11 Sub-task Resolved William Hyun  
          3.
          Upgrade checkstyle plugins Sub-task Resolved William Hyun  
          4.
          Upgrade ORC to 1.7.1 Sub-task Resolved William Hyun  
          5.
          Move Java 17 on GitHub Actions from EA to LTS release Sub-task Resolved Kousuke Saruta  
          6.
          Exclude hadoop-client-api transitive dependency from orc-core Sub-task Resolved Kousuke Saruta  
          7.
          Support K8s integration tests for Java 17 Sub-task Resolved Kousuke Saruta  
          8.
          Change HiveExternalCatalogVersionsSuite to be able to test with Java 17 Sub-task Resolved Kousuke Saruta  
          9.
          Fix Invalid value for HourOfAmPm when testing on JDK 14 Sub-task Resolved Yuming Wang  
          10.
          Upgrade ZooKeeper to 3.6.2 Sub-task Resolved Yuming Wang  
          11.
          Add java.security.jgss/sun.security.krb5 to DEFAULT_MODULE_OPTIONS Sub-task Resolved Yuming Wang  
          12.
          Upgrade Scala to 2.12.14 Sub-task Resolved Dongjoon Hyun  
          13.
          Upgrade Scala to 2.12.15 Sub-task Resolved Dongjoon Hyun  
          14.
          Upgrade ORC to 1.7.0 Sub-task Resolved Dongjoon Hyun  
          15.
          Add Daily GitHub Action jobs for Java11/17 Sub-task Resolved Dongjoon Hyun  
          16.
          Remove explicit GC options from Scala tests Sub-task Resolved Dongjoon Hyun  
          17.
          Add `.java-version` to `.gitignore` and `.rat-excludes` Sub-task Resolved Dongjoon Hyun  
          18.
          Support K8s image building with Java 17 Sub-task Resolved Dongjoon Hyun  
          19.
          Add Java17 PySpark daily test coverage Sub-task Resolved Dongjoon Hyun  
          20.
          Add Java17 SparkR daily test coverage Sub-task Resolved Dongjoon Hyun  
          21.
          Document Java 17 support Sub-task Resolved Dongjoon Hyun  
          22.
          Upgrade Maven to 3.8.4 Sub-task Resolved Dongjoon Hyun  
          23.
          Upgrade SBT to 1.6.0 Sub-task Resolved Dongjoon Hyun  
          24.
          Set file.encoding to UTF-8 for SBT tests Sub-task Resolved Dongjoon Hyun  
          25.
          Upgrade Scala 2.13 to 2.13.7 Sub-task Resolved Yang Jie  
          26.
          Make dev/mima runs on Java 17 Sub-task Resolved Yang He  
          27.
          Make sql/core and dependent modules all UTs pass on Java 17 Sub-task Resolved Yang Jie  
          28.
          "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17 Sub-task Resolved Yang Jie  
          29.
          Manual disabled format `B` for `date_format` function to compatibility with Java 8 behavior. Sub-task Resolved Yang Jie  
          30.
          WholeStageCodegenSparkSubmitSuite Failed with Java 17 Sub-task Resolved Yang Jie  
          31.
          `select format_string('%0$s', 'Hello')` has different behavior when using java 8 and Java 17 Sub-task Resolved Yang Jie  
          32.
          Pass all UTs in `mllib-local` and `mllib` with Java 17 Sub-task Resolved Yang Jie  
          33.
          Pass all UTs in `repl` with Java 17 Sub-task Resolved Yang Jie  
          34.
          Pass all UTs in `external/avro` with Java 17 Sub-task Resolved Yang Jie  
          35.
          Pass all UTs in `sql/hive` with Java 17 Sub-task Resolved Yang Jie  
          36.
          Pass all UTs in `yarn` with Java 17 Sub-task Resolved Yang Jie  
          37.
          Supplement all micro benchmark results use to Java 17 Sub-task Resolved Yang Jie  
          38.
          Upgrade RoaringBitmap to 0.9.23 Sub-task Resolved Yang Jie  
          39.
          Add InMemoryColumnarBenchmark and StateStoreBasicOperationsBenchmark Java 11/17 result Sub-task Resolved Yang Jie  
          40.
          Try to remove `--add-exports` compile option for Java 17 Sub-task Resolved Yang Jie  
          41.
          Add Java 17 support for spark pull request builds Sub-task Closed Unassigned  
          42.
          Add a vectorized BLAS implementation Sub-task Resolved Ludovic Henry  
          43.
          Support Java 17 in SparkR SystemRequirements Sub-task Resolved Darek

          0%

          Original Estimate - 1h
          Remaining Estimate - 1h
          44.
          Support Java 17 in `dev/test-dependencies.sh` Sub-task Closed Unassigned  
          45.
          Adapt uses of JDK 17 Internal APIs Sub-task Closed Unassigned  
          46.
          Upgrade Kafka to 3.1.0 Sub-task In Progress Dongjoon Hyun  

          Activity

            People

              LuciferYang Yang Jie
              dongjoon Dongjoon Hyun
              Votes:
              3 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 1h
                  1h
                  Remaining:
                  Remaining Estimate - 1h
                  1h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified