Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3222

DFSInputStream#openInfo should not silently get the length as 0 when locations length is zero for last partial block.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.3, 2.0.0-alpha, 3.0.0
    • Fix Version/s: 2.0.0-alpha
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      I have seen one situation with Hbase cluster.

      Scenario is as follows:

      1)1.5 blocks has been written and synced.

      2)Suddenly cluster has been restarted.

      Reader opened the file and trying to get the length., By this time partial block contained DNs are not reported to NN. So, locations for this partial block would be 0. In this case, DFSInputStream assumes that, 1 block size as final size.

      But reader also assuming that, 1 block size is the final length and setting his end marker. Finally reader ending up reading only partial data. Due to this, HMaster could not replay the complete edits.

      Actually this happend with 20 version. Looking at the code, same should present in trunk as well.

          int replicaNotFoundCount = locatedblock.getLocations().length;
          
          for(DatanodeInfo datanode : locatedblock.getLocations()) {
      ..........
      ..........
       // Namenode told us about these locations, but none know about the replica
          // means that we hit the race between pipeline creation start and end.
          // we require all 3 because some other exception could have happened
          // on a DN that has it.  we want to report that error
          if (replicaNotFoundCount == 0) {
            return 0;
          }
      
      
      1. HDFS-3222-Test.patch
        3 kB
        Uma Maheswara Rao G
      2. HDFS-3222.patch
        7 kB
        Uma Maheswara Rao G
      3. HDFS-3222.patch
        8 kB
        Uma Maheswara Rao G

        Issue Links

          Activity

          Uma Maheswara Rao G created issue -
          Uma Maheswara Rao G made changes -
          Field Original Value New Value
          Link This issue is related to HBASE-5666 [ HBASE-5666 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HBASE-5666 [ HBASE-5666 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HBASE-5745 [ HBASE-5745 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-3222-Test.patch [ 12521942 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-3222.patch [ 12523528 ]
          Uma Maheswara Rao G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 3.0.0, 2.0.0 [ 12320356, 12320353 ] 2.0.0, 3.0.0 [ 12320353, 12320356 ]
          Uma Maheswara Rao G made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 3.0.0, 2.0.0 [ 12320356, 12320353 ] 2.0.0, 3.0.0 [ 12320353, 12320356 ]
          Uma Maheswara Rao G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 3.0.0, 2.0.0 [ 12320356, 12320353 ] 2.0.0, 3.0.0 [ 12320353, 12320356 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-3222.patch [ 12524471 ]
          Uma Maheswara Rao G made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 3.0.0, 2.0.0 [ 12320356, 12320353 ] 2.0.0, 3.0.0 [ 12320353, 12320356 ]
          Uma Maheswara Rao G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 3.0.0, 2.0.0 [ 12320356, 12320353 ] 2.0.0, 3.0.0 [ 12320353, 12320356 ]
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 3.0.0, 2.0.0 [ 12320356, 12320353 ] 2.0.0, 3.0.0 [ 12320353, 12320356 ]
          Uma Maheswara Rao G made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Target Version/s 3.0.0, 2.0.0 [ 12320356, 12320353 ] 2.0.0, 3.0.0 [ 12320353, 12320356 ]
          Fix Version/s 2.0.0 [ 12320353 ]
          Fix Version/s 3.0.0 [ 12320356 ]
          Resolution Fixed [ 1 ]
          Jimmy Xiang made changes -
          Link This issue is related to HBASE-6443 [ HBASE-6443 ]
          Uma Maheswara Rao G made changes -
          Link This issue relates to HDFS-3701 [ HDFS-3701 ]
          Todd Lipcon made changes -
          Link This issue is related to HDFS-4380 [ HDFS-4380 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HDFS-4516 [ HDFS-4516 ]
          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320356 ]

            People

            • Assignee:
              Uma Maheswara Rao G
              Reporter:
              Uma Maheswara Rao G
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development