[FLUME-3146] Use public API HdfsDataOutputStream#getCurrentBlockReplication where applicable

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Assignee: Peter Turcsanyi
• Reporter: Wei-Chiu Chuang
• Created: 14/Aug/17 22:47
• Updated: 02/Oct/19 17:15

Description

AbstractHDFSWrite#reflectGetNumCurrentReplicas uses a private HDFS API, which causes it to emit an confusing error message:

2017-08-08 22:01:10,743 INFO org.apache.flume.sink.hdfs.AbstractHDFSWriter: FileSystem's output stream doesn't support getNumCurrentReplicas; --HDFS-826 not available; fsOut=org.apache.hadoop.crypto.CryptoOutputStream; err=java.lang.NoSuchMethodException: org.apache.hadoop.crypto.CryptoOutputStream.getNumCurrentReplicas()
2017-08-08 22:02:11,086 ERROR org.apache.flume.sink.hdfs.AbstractHDFSWriter: Error while trying to hflushOrSync!

This error message is emitted even with the most recent version of HDFS that has HDFS-826 for sure.

Turns out that the relevant code is not valid with HDFS encryption. Instead of using the current private (and deprecated) getNumCurrentReplicas#getNumCurrentReplicas() API, it should use HdfsDataOutputStream#getCurrentBlockReplication, which is a public API and which considers the case of HDFS encryption.

Issue Links

• links to: GitHub Pull Request #250