Details
-
Task
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
AbstractHDFSWriter uses some HDFS client calls through reflection in order to retrieve the default / current replication count and also to determine if the client supports hflush() or only sync().
This reflection magic comes from the the early versions of Hadoop and it is quite obsolete now because Hadoop 2+ provides public API for the these calls and supports hflush().
So the reflection calls could be eliminated now.
Furthermore, the whole replication count checking logic seems to be a bit overkill, because it should be handled by HDFS itself and not in the client code.