[HDFS-2834] ByteBuffer-based read API for DFSInputStream

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.2-alpha
• Component/s: hdfs-client, performance
• Hadoop Flags: Reviewed
• Assignee: Henry Robinson
• Reporter: Henry Robinson
• Created: 24/Jan/12 19:36
• Updated: 24/Nov/20 14:29
• Resolved: 21/Mar/12 17:31

Description

The DFSInputStream read-path always copies bytes into a JVM-allocated byte[]. Although for many clients this is desired behaviour, in certain situations, such as native-reads through libhdfs, this imposes an extra copy penalty since the byte[] needs to be copied out again into a natively readable memory area.

For these cases, it would be preferable to allow the client to supply its own buffer, wrapped in a ByteBuffer, to avoid that final copy overhead.

Issue Links

• breaks: HDFS-3243 TestParallelRead timing out on jenkins
• depends upon: HADOOP-8135 Add ByteBufferReadable interface to FSDataInputStream
• is depended upon by: HDFS-3110 libhdfs implementation of direct read API
• is related to: HBASE-8143 HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
• is related to: HDFS-3053 Support for true zero-copy (mmap-based) reads
• is related to: HDFS-3051 A zero-copy ScatterGatherRead api from FSDataInputStream
• is related to: HBASE-21879 Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose
• relates to: HADOOP-8148 Zero-copy ByteBuffer-based compressor / decompressor API
• relates to: HDFS-15693 Improve native code's performance when writing to HDFS
• relates to: HDFS-3246 pRead equivalent for direct read path