Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10047

Add a directbuffer Decompressor API to hadoop

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.3.0
    • io
    • Direct Bytebuffer decompressors for Zlib (Deflate & Gzip) and Snappy

    Description

      With the Zero-Copy reads in HDFS (HDFS-5260), it becomes important to perform all I/O operations without copying data into byte[] buffers or other buffers which wrap over them.

      This is a proposal for adding a DirectDecompressor interface to the io.compress, to indicate codecs which want to surface the direct buffer layer upwards.

      The implementation should work with direct heap/mmap buffers and cannot assume .array() availability.

      Attachments

        1. decompress-benchmark.tgz
          2 kB
          Gopal Vijayaraghavan
        2. DirectCompressor.html
          13 kB
          Gopal Vijayaraghavan
        3. DirectDecompressor.html
          13 kB
          Gopal Vijayaraghavan
        4. HADOOP-10047-final.patch
          26 kB
          Gopal Vijayaraghavan
        5. HADOOP-10047-redo-WIP.patch
          8 kB
          Gopal Vijayaraghavan
        6. HADOOP-10047-trunk.patch
          24 kB
          Gopal Vijayaraghavan
        7. HADOOP-10047-WIP.patch
          19 kB
          Gopal Vijayaraghavan
        8. HADOOP-10047-with-tests.patch
          22 kB
          Gopal Vijayaraghavan

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6347 ZeroCopy read path for ORC RecordReader

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-10116 fix "inconsistent synchronization" warnings in ZlibCompressor

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-8148 Zero-copy ByteBuffer-based compressor / decompressor API

          • Major - Major loss of function.
          • Open

          Activity

            People

              gopalv Gopal Vijayaraghavan
              gopalv Gopal Vijayaraghavan
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: