[HDFS-14083] libhdfs logs errors when opened FS doesn't support ByteBufferReadable

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 3.0.3
• Component/s: libhdfs, native
• Release Note: Made changes to TestClientMetrics.java to fix the issue.
• Assignee: Pranay Singh
• Reporter: Pranay Singh
• Created: 13/Nov/18 19:39
• Updated: 01/Mar/19 00:36
• Resolved: 01/Mar/19 00:36

Description

Problem:
------------
There is excessive error logging when a file is opened by libhdfs (DFSClient/HDFS) in S3 environment, this issue is caused because buffered read is not supported in S3 environment, HADOOP-14603 "S3A input stream to support ByteBufferReadable"

The following message is printed repeatedly in the error log/ to STDERR:

--------------------------------------------------------------------------------------------------
UnsupportedOperationException: Byte-buffer read unsupported by input streamjava.lang.UnsupportedOperationException: Byte-buffer read unsupported by input stream
        at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:150)

Root cause

After investigating the issue, it appears that the above exception is printed because
when a file is opened via hdfsOpenFileImpl() calls readDirect() which is hitting this
exception.

Fix:

Since the hdfs client is not initiating the byte buffered read but is happening in a implicit manner, we should not be generating the error log during open of a file.

Issue Links

• is related to: HDFS-14111 hdfsOpenFile on HDFS causes unnecessary IO from file offset 0