Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10047

Add a directbuffer Decompressor API to hadoop

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.3.0
    • io
    • Direct Bytebuffer decompressors for Zlib (Deflate & Gzip) and Snappy

    Description

      With the Zero-Copy reads in HDFS (HDFS-5260), it becomes important to perform all I/O operations without copying data into byte[] buffers or other buffers which wrap over them.

      This is a proposal for adding a DirectDecompressor interface to the io.compress, to indicate codecs which want to surface the direct buffer layer upwards.

      The implementation should work with direct heap/mmap buffers and cannot assume .array() availability.

      Attachments

        1. decompress-benchmark.tgz
          2 kB
          Gopal Vijayaraghavan
        2. DirectCompressor.html
          13 kB
          Gopal Vijayaraghavan
        3. DirectDecompressor.html
          13 kB
          Gopal Vijayaraghavan
        4. HADOOP-10047-final.patch
          26 kB
          Gopal Vijayaraghavan
        5. HADOOP-10047-redo-WIP.patch
          8 kB
          Gopal Vijayaraghavan
        6. HADOOP-10047-trunk.patch
          24 kB
          Gopal Vijayaraghavan
        7. HADOOP-10047-WIP.patch
          19 kB
          Gopal Vijayaraghavan
        8. HADOOP-10047-with-tests.patch
          22 kB
          Gopal Vijayaraghavan

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            gopalv Gopal Vijayaraghavan
            gopalv Gopal Vijayaraghavan
            Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment