Details
-
Bug
-
Status: Patch Available
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
AbstractHDFSWrite#reflectGetNumCurrentReplicas uses a private HDFS API, which causes it to emit an confusing error message:
2017-08-08 22:01:10,743 INFO org.apache.flume.sink.hdfs.AbstractHDFSWriter: FileSystem's output stream doesn't support getNumCurrentReplicas; --HDFS-826 not available; fsOut=org.apache.hadoop.crypto.CryptoOutputStream; err=java.lang.NoSuchMethodException: org.apache.hadoop.crypto.CryptoOutputStream.getNumCurrentReplicas() 2017-08-08 22:02:11,086 ERROR org.apache.flume.sink.hdfs.AbstractHDFSWriter: Error while trying to hflushOrSync!
This error message is emitted even with the most recent version of HDFS that has HDFS-826 for sure.
Turns out that the relevant code is not valid with HDFS encryption. Instead of using the current private (and deprecated) getNumCurrentReplicas#getNumCurrentReplicas() API, it should use HdfsDataOutputStream#getCurrentBlockReplication, which is a public API and which considers the case of HDFS encryption.
Attachments
Attachments
Issue Links
- links to