[HDFS-14306] Re-use Java buffers across libhdfs read calls

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: hdfs-client, libhdfs, native
• Assignee: Sahil Takiar
• Reporter: Sahil Takiar
• Created: 21/Feb/19 18:45
• Updated: 21/Feb/19 18:45

Description

Filesystem connectors that don't natively support ByteBuffer cannot fully take advantage of the direct read optimizations added in HDFS-2834 / HDFS-3110. In this case, they fall back to the regular read path which allocates a Java byte array for every read. This leads to a lot of perf overhead wasted in TypeArrayKlass::allocate_common.

A performance optimization would be to re-use Java byte arrays between calls to hdfsRead / hdfsPread so that new arrays don't need to be allocate for every single call.