[HDFS-3222] DFSInputStream#openInfo should not silently get the length as 0 when locations length is zero for last partial block.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.3, 2.0.0-alpha, 3.0.0-alpha1
• Fix Version/s: 2.0.0-alpha
• Component/s: hdfs-client
• Target Version/s: 2.0.0-alpha
• Hadoop Flags: Reviewed
• Assignee: Uma Maheswara Rao G
• Reporter: Uma Maheswara Rao G
• Created: 06/Apr/12 18:49
• Updated: 12/May/16 18:18
• Resolved: 26/Apr/12 20:27

Description

I have seen one situation with Hbase cluster.

Scenario is as follows:

1)1.5 blocks has been written and synced.

2)Suddenly cluster has been restarted.

Reader opened the file and trying to get the length., By this time partial block contained DNs are not reported to NN. So, locations for this partial block would be 0. In this case, DFSInputStream assumes that, 1 block size as final size.

But reader also assuming that, 1 block size is the final length and setting his end marker. Finally reader ending up reading only partial data. Due to this, HMaster could not replay the complete edits.

Actually this happend with 20 version. Looking at the code, same should present in trunk as well.

    int replicaNotFoundCount = locatedblock.getLocations().length;
    
    for(DatanodeInfo datanode : locatedblock.getLocations()) {
..........
..........
 // Namenode told us about these locations, but none know about the replica
    // means that we hit the race between pipeline creation start and end.
    // we require all 3 because some other exception could have happened
    // on a DN that has it.  we want to report that error
    if (replicaNotFoundCount == 0) {
      return 0;
    }

Issue Links

• is related to: HDFS-4380 Opening a file for read before writer writes a block causes NPE
• is related to: HBASE-5745 SequenceFileLogReader#getPos may get wrong length on DFS restart
• is related to: HDFS-4516 Client crash after block allocation and NN switch before lease recovery for the same file can cause readers to fail forever
• is related to: HBASE-6443 HLogSplitter should ignore 0 length files
• relates to: HDFS-3701 HDFS may miss the final block when reading a file opened for writing if one of the datanode is dead