Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35781

Support Spark on Apple Silicon on macOS natively on Java 17

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • Build

    Description

      This is an umbrella JIRA tracking the progress of supporting Apple Silicon on macOS natively.

      Attachments

        Issue Links

          is related to

          Umbrella - An overarching type made of sub-tasks SPARK-38396 Improve K8s Integration Tests

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-35782 leveldbjni doesn't work in Apple Silicon on macOS

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. SPARK-36731 BrotliCodec doesn't support Apple Silicon on MacOS

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. SPARK-33772 Build and Run Spark on Java 17

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-38888 Add `RocksDBProvider` similar to `LevelDBProvider`

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          1.
          		 Upgrade SBT to 1.5.4 Sub-task Resolved Dongjoon Hyun
          2.
          		 Convert LevelDBSuite.IntKeyType from a nested class to a normal class Sub-task Resolved Dongjoon Hyun
          3.
          		 Add `ExtendedRocksDBTest` and disable RocksDB tests on Apple Silicon Sub-task Resolved Dongjoon Hyun
          4.
          		 Add ExtendedLevelDBTest and disable LevelDB tests on Apple Silicon Sub-task Resolved Dongjoon Hyun
          5.
          		 Fix MultilayerPerceptronClassifierTest.test_raw_and_probability_prediction Sub-task Resolved Dongjoon Hyun
          6.
          		 Reduce weights in GaussianMixtureSuite Sub-task Resolved Dongjoon Hyun
          7.
          		 Add RocksDB Implementation for KVStore Sub-task Resolved Dongjoon Hyun
          8.
          		 Support RocksDB backend in Spark History Server Sub-task Resolved Dongjoon Hyun
          9.
          		 Optimize RocksDB KVStore configurations Sub-task Resolved Dongjoon Hyun
          10.
          		 Enable Apple Silicon Jenkins CI (Java/Scala/Python/R) Sub-task Resolved Dongjoon Hyun
          11.
          		 Support ARM64 in Java 17 docker image Sub-task Resolved Dongjoon Hyun
          12.
          		 Use Java 17 in K8s integration tests Sub-task Resolved Dongjoon Hyun
          13.
          		 HistoryServerDiskManager.appStorePath should use backend-based extensions Sub-task Resolved Dongjoon Hyun
          14.
          		 Upgrade lz4-java to 1.8.0 Sub-task Resolved Kousuke Saruta
          15.
          		 Upgrade Netty from 4.1.63 to 4.1.68 Sub-task Resolved Kousuke Saruta
          16.
          		 UT `handles k8s cluster mode` fails on MacOs >= 10.15 Sub-task Resolved Yazhi Wang
          17.
          		 Cannot run leveldb related UTs on Mac OS of M1 architecture Sub-task Closed Unassigned
          18.
          		 Refactor KVUtils and add UTs related to RocksDB Sub-task Resolved Yang Jie
          19.
          		 Upgrade rocksdbjni to 7.0.3 Sub-task Resolved Yang Jie
          20.
          		 Enable RocksDB tests on Apple Silicon on MacOS Sub-task Resolved Dongjoon Hyun

          Activity

            People

              dongjoon Dongjoon Hyun
              dbtsai DB Tsai
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: