Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34651

Improve ZSTD support

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.2.0
    • Fix Version/s: None
    • Component/s: Spark Core, SQL
    • Labels:
      None
    • Target Version/s:

      Attachments

      Issue Links

      1.
      Upgrade zstd-jni to 1.4.8-3 Sub-task Resolved William Hyun Actions
      2.
      Upgrade ORC to 1.6.6 Sub-task Resolved Dongjoon Hyun Actions
      3.
      Update ORC data source documentation Sub-task Resolved Dongjoon Hyun Actions
      4.
      Add ZStandardBenchmark Sub-task Resolved Dongjoon Hyun Actions
      5.
      Support ZSTD JNI BufferPool Sub-task Resolved Dongjoon Hyun Actions
      6.
      Support ZSTD compression in ORC data source Sub-task Resolved Dongjoon Hyun Actions
      7.
      Upgrade ZSTD-JNI to 1.4.8-5 to API compatibility Sub-task Resolved Dongjoon Hyun Actions
      8.
      Add zstandard codec to spark.sql.avro.compression.codec Sub-task Resolved Yuming Wang Actions
      9.
      Enable Zstandard buffer pool by default Sub-task Resolved Dongjoon Hyun Actions
      10.
      Upgrade to ZSTD JNI 1.4.8-6 Sub-task Resolved Dongjoon Hyun Actions
      11.
      Use zstd for spark.eventLog.compression.codec by default Sub-task Resolved Dongjoon Hyun Actions
      12.
      Exclude Avro's transitive zstd-jni dependency Sub-task Resolved Dongjoon Hyun Actions
      13.
      Exclude zstd-jni transitive dependency from Kafka Client Sub-task Resolved Dongjoon Hyun Actions
      14.
      Upgrade ZSTD-JNI to 1.4.8-7 and use NoFinalizer classes Sub-task Resolved Dongjoon Hyun Actions
      15.
      Upgrade ZSTD-JNI to 1.4.9-1 Sub-task Resolved Dongjoon Hyun Actions
      16.
      Support ZSTD compression in Parquet data sources Sub-task Resolved Unassigned Actions
      17.
      Add Parquet ZSTD compression test coverage Sub-task Resolved Dongjoon Hyun Actions
      18.
      Use zstd codec name in ORC file names Sub-task Resolved Dongjoon Hyun Actions
      19.
      Use zstd for spark.io.compression.codec by default Sub-task In Progress Unassigned Actions
      20.
      Upgrade Kafka to 2.8.0 Sub-task Resolved Dongjoon Hyun Actions
      21.
      Upgrade ZSTD-JNI to 1.5.0-2 Sub-task Resolved David Christle Actions

        Activity

          People

          • Assignee:
            dongjoon Dongjoon Hyun
            Reporter:
            dongjoon Dongjoon Hyun

            Dates

            • Created:
              Updated:

              Issue deployment