[HDFS-14111] hdfsOpenFile on HDFS causes unnecessary IO from file offset 0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: hdfs-client, libhdfs
• Assignee: Sahil Takiar
• Reporter: Todd Lipcon
• Created: 28/Nov/18 18:18
• Updated: 27/Aug/20 17:57
• Resolved: 06/Mar/19 23:00

Description

hdfsOpenFile() calls readDirect() with a 0-length argument in order to check whether the underlying stream supports bytebuffer reads. With DFSInputStream, the read(0) isn't short circuited, and results in the DFSClient opening a block reader. In the case of a remote block, the block reader will actually issue a read of the whole block, causing the datanode to perform unnecessary IO and network transfers in order to fill up the client's TCP buffers. This causes performance degradation.

Issue Links

• is related to: HDFS-14483 Backport HDFS-14111,HDFS-3246 ByteBuffer pread interface to branch-2.9
• relates to: HDFS-8160 Long delays when calling hdfsOpenFile()
• relates to: HADOOP-15229 Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.
• relates to: HADOOP-15691 Add PathCapabilities to FS and FC to complement StreamCapabilities
• relates to: HDFS-14083 libhdfs logs errors when opened FS doesn't support ByteBufferReadable