[HADOOP-10047] Add a directbuffer Decompressor API to hadoop

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: io
• Labels: compression
• Release Note: Direct Bytebuffer decompressors for Zlib (Deflate & Gzip) and Snappy
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 15/Oct/13 14:37
• Updated: 24/Feb/14 20:56
• Resolved: 19/Nov/13 15:40

Description

With the Zero-Copy reads in HDFS (HDFS-5260), it becomes important to perform all I/O operations without copying data into byte[] buffers or other buffers which wrap over them.

This is a proposal for adding a DirectDecompressor interface to the io.compress, to indicate codecs which want to surface the direct buffer layer upwards.

The implementation should work with direct heap/mmap buffers and cannot assume .array() availability.

Issue Links

• blocks: HIVE-6347 ZeroCopy read path for ORC RecordReader
• is related to: HADOOP-10116 fix "inconsistent synchronization" warnings in ZlibCompressor
• is related to: HADOOP-8148 Zero-copy ByteBuffer-based compressor / decompressor API