Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.5.0, 2.6.0
-
None
-
None
Description
I recently have come across a bug that causes an infinite loop in the DFSClient. I have experienced this issue in hadoop 2.5.0 and the issue seems to be present in 2.6.0.
The bug is hard to reproduce, it seems to only occurs when the NameNode is under great pressure because I think it's a timing issue.
On the client side, a small file (100s of bytes or so) is written and then sync() is called. The depreciated sync because the code is setup to cross compile against hadoop 1 and hadoop 2. After the sync is called the close happens on the outputstream in another thread async to the writing thread. This happens because the close call can be very time consuming.
Once the sync happens and the outputstream is handed off to the closing thread. The writing thread turns around and reads the output it has written and synced. When this happens I believe the client reads the length from the Namenode which appears to still be 0 (more on that in a moment).
Once the inputstream is open and the first byte is trying to be read the DFSInputStream goes into an infinite loop. It appears to be error handling logical that is not handling all IOExceptions.
fetchBlockByteRange => https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L991
The loop occurs in the fetchBlockByteRange method, which catches all IOExceptions and just recalls the actualGetFromOneDataNode method, assuming that method handles everything correctly.
actualGetFromOneDataNode => https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1025
In the actualGetFromOneDataNode inside the while loop it calls getBlockAt which throws a IOException that is not handled by the actualGetFromOneDataNode method (the real issue).
actualGetFromOneDataNode calls getBlockAt =>
https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1040
In the getBlockAt method it checks that position to read are within the file length, which I believe to still be zero at this point. This is where I believe the IOException is thrown.
And because the IOException is not handled in the actualGetFromOneDataNode method and the fetchBlockByteRange blindly recalls the actualGetFromOneDataNode method over and over again the infinite loop is created.
My current work around is to wait until the file length is properly reported by the namenode before opening the file. Likely this is the correct choice regardless, but I think that client should never go into an infinite loop during an error condition.