Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21276

Update lz4-java to remove custom LZ4BlockInputStream

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.1.1, 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      We currently use custom LZ4BlockInputStream to read concatenated byte stream in shuffle (https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/io/LZ4BlockInputStream.java#L38). In the recent pr (https://github.com/lz4/lz4-java/pull/105), this functionality is implemented even in lz4-java upstream. So, we might update the lz4-java package that will be released in near future.

      Issue about the next lz4-java release
      https://github.com/lz4/lz4-java/issues/98

      Diff between the latest release and the master in lz4-java
      https://github.com/lz4/lz4-java/compare/62f7547abb0819d1ca1e669645ee1a9d26cd60b0...6480bd9e06f92471bf400c16d4d5f3fd2afa3b3d

      • fixed NPE in XXHashFactory similarly
      • Don't place resources in default package to support shading
      • Fixes ByteBuffer methods failing to apply arrayOffset() for array-backed
      • Try to load lz4-java from java.library.path, then fallback to bundled
      • Add ppc64le binary
      • Add s390x JNI binding
      • Add basic LZ4 Frame v1.5.0 support
      • enable aarch64 support for lz4-java
      • Allow unsafeInstance() for ppc64le archiecture
      • Add unsafeInstance support for AArch64
      • Support 64-bit JNI build on Solaris
      • Avoid over-allocating a buffer
      • Allow EndMark to be incompressible for LZ4FrameInputStream.
      • Concat byte stream

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                maropu Takeshi Yamamuro
                Reporter:
                maropu Takeshi Yamamuro
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: