Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34651

Improve ZSTD support

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • Spark Core, SQL

    Attachments

      Issue Links

      1.
      Upgrade zstd-jni to 1.4.8-3 Sub-task Resolved William Hyun Actions
      2.
      Upgrade ORC to 1.6.6 Sub-task Resolved Dongjoon Hyun Actions
      3.
      Update ORC data source documentation Sub-task Resolved Dongjoon Hyun Actions
      4.
      Add ZStandardBenchmark Sub-task Resolved Dongjoon Hyun Actions
      5.
      Support ZSTD JNI BufferPool Sub-task Resolved Dongjoon Hyun Actions
      6.
      Support ZSTD compression in ORC data source Sub-task Resolved Dongjoon Hyun Actions
      7.
      Upgrade ZSTD-JNI to 1.4.8-5 to API compatibility Sub-task Resolved Dongjoon Hyun Actions
      8.
      Add zstandard codec to spark.sql.avro.compression.codec Sub-task Resolved Yuming Wang Actions
      9.
      Enable Zstandard buffer pool by default Sub-task Resolved Dongjoon Hyun Actions
      10.
      Upgrade to ZSTD JNI 1.4.8-6 Sub-task Resolved Dongjoon Hyun Actions
      11.
      Use zstd for spark.eventLog.compression.codec by default Sub-task Resolved Dongjoon Hyun Actions
      12.
      Exclude Avro's transitive zstd-jni dependency Sub-task Resolved Dongjoon Hyun Actions
      13.
      Exclude zstd-jni transitive dependency from Kafka Client Sub-task Resolved Dongjoon Hyun Actions
      14.
      Upgrade ZSTD-JNI to 1.4.8-7 and use NoFinalizer classes Sub-task Resolved Dongjoon Hyun Actions
      15.
      Upgrade ZSTD-JNI to 1.4.9-1 Sub-task Resolved Dongjoon Hyun Actions
      16.
      Support ZSTD compression in Parquet data sources Sub-task Resolved Unassigned Actions
      17.
      Add Parquet ZSTD compression test coverage Sub-task Resolved Dongjoon Hyun Actions
      18.
      Upgrade Kafka to 2.8.0 Sub-task Resolved Dongjoon Hyun Actions
      19.
      Upgrade ZSTD-JNI to 1.5.0-2 Sub-task Resolved David Christle Actions
      20.
      Upgrade aircompressor to 1.21 Sub-task Resolved Dongjoon Hyun Actions
      21.
      Use zstd codec name in ORC file names Sub-task Resolved Dongjoon Hyun Actions
      22.
      Change default spark.io.compression.codec to zstd Sub-task In Progress Unassigned Actions

      Activity

        This comment will be Viewable by All Users Viewable by All Users
        Cancel

        People

          dongjoon Dongjoon Hyun
          dongjoon Dongjoon Hyun
          Votes:
          0 Vote for this issue
          Watchers:
          6 Start watching this issue

          Dates

            Created:
            Updated:
            Resolved:

            Slack

              Issue deployment