Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10047

Add a directbuffer Decompressor API to hadoop

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.3.0
    • io
    • Direct Bytebuffer decompressors for Zlib (Deflate & Gzip) and Snappy

    Description

      With the Zero-Copy reads in HDFS (HDFS-5260), it becomes important to perform all I/O operations without copying data into byte[] buffers or other buffers which wrap over them.

      This is a proposal for adding a DirectDecompressor interface to the io.compress, to indicate codecs which want to surface the direct buffer layer upwards.

      The implementation should work with direct heap/mmap buffers and cannot assume .array() availability.

      Attachments

        1. DirectCompressor.html
          13 kB
          Gopal Vijayaraghavan
        2. DirectDecompressor.html
          13 kB
          Gopal Vijayaraghavan
        3. HADOOP-10047-WIP.patch
          19 kB
          Gopal Vijayaraghavan
        4. HADOOP-10047-with-tests.patch
          22 kB
          Gopal Vijayaraghavan
        5. HADOOP-10047-redo-WIP.patch
          8 kB
          Gopal Vijayaraghavan
        6. HADOOP-10047-final.patch
          26 kB
          Gopal Vijayaraghavan
        7. HADOOP-10047-trunk.patch
          24 kB
          Gopal Vijayaraghavan
        8. decompress-benchmark.tgz
          2 kB
          Gopal Vijayaraghavan

        Issue Links

          Activity

            People

              gopalv Gopal Vijayaraghavan
              gopalv Gopal Vijayaraghavan
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: