Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2815

Namenode is not coming out of safemode when we perform ( NN crash + restart ) . Also FSCK report shows blocks missed.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.22.0, 0.23.1, 1.0.0, 1.1.0, 2.0.0-alpha
    • Fix Version/s: 1.1.1, 0.23.2
    • Component/s: namenode
    • Labels:
      None

      Description

      When tested the HA(internal) with continuous switch with some 5mins gap, found some blocks missed and namenode went into safemode after next switch.

      After the analysis, i found that this files already deleted by clients. But i don't see any delete commands logs namenode log files. But namenode added that blocks to invalidateSets and DNs deleted the blocks.
      When restart of the namenode, it went into safemode and expecting some more blocks to come out of safemode.

      Here the reason could be that, file has been deleted in memory and added into invalidates after this it is trying to sync the edits into editlog file. By that time NN asked DNs to delete that blocks. Now namenode shuts down before persisting to editlogs.( log behind)
      Due to this reason, we may not get the INFO logs about delete, and when we restart the Namenode (in my scenario it is again switch), Namenode expects this deleted blocks also, as delete request is not persisted into editlog before.

      I reproduced this scenario with bedug points. I feel, We should not add the blocks to invalidates before persisting into Editlog.

      Note: for switch, we used kill -9 (force kill)

      I am currently in 0.20.2 version. Same verified in 0.23 as well in normal crash + restart scenario.

      1. HDFS-2815.patch
        1 kB
        Uma Maheswara Rao G
      2. HDFS-2815.patch
        1 kB
        Uma Maheswara Rao G
      3. HDFS-2815-22-branch.patch
        1 kB
        Uma Maheswara Rao G
      4. HDFS-2815-branch-1.patch
        1 kB
        Uma Maheswara Rao G
      5. HDFS-2815-Branch-1.patch
        8 kB
        Uma Maheswara Rao G

        Issue Links

          Activity

          Uma Maheswara Rao G created issue -
          Uma Maheswara Rao G made changes -
          Field Original Value New Value
          Attachment HDFS-2815.patch [ 12511806 ]
          Uma Maheswara Rao G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Link This issue is broken by HDFS-173 [ HDFS-173 ]
          Uma Maheswara Rao G made changes -
          Component/s name-node [ 12312926 ]
          Uma Maheswara Rao G made changes -
          Target Version/s 0.24.0, 0.23.2 [ 12317653, 12319852 ]
          Suresh Srinivas made changes -
          Affects Version/s 1.1.0 [ 12317959 ]
          Target Version/s 0.23.2, 0.24.0 [ 12319852, 12317653 ] 0.24.0, 0.23.2 [ 12317653, 12319852 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-2815.patch [ 12514253 ]
          Suresh Srinivas made changes -
          Fix Version/s 0.24.0 [ 12317653 ]
          Fix Version/s 0.23.2 [ 12319852 ]
          Target Version/s 0.23.2, 0.24.0 [ 12319852, 12317653 ] 0.24.0, 1.1.0, 0.23.2 [ 12317653, 12317959, 12319852 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-2815-Branch-1.patch [ 12517008 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-2815-22-branch.patch [ 12517041 ]
          Robert Joseph Evans made changes -
          Target Version/s 0.23.2, 1.1.0, 0.24.0 [ 12319852, 12317959, 12317653 ] 1.1.0, 2.0.0, 3.0.0 [ 12317959, 12320353, 12320356 ]
          Matt Foley made changes -
          Fix Version/s 3.0.0 [ 12320356 ]
          Fix Version/s 2.0.0-alpha [ 12320353 ]
          Fix Version/s 0.24.0 [ 12317653 ]
          Fix Version/s 0.23.2 [ 12319852 ]
          Target Version/s 1.1.0, 2.0.0-alpha, 3.0.0 [ 12317959, 12320353, 12320356 ] 2.0.0-alpha, 1.2.0, 3.0.0 [ 12320353, 12321657, 12320356 ]
          Uma Maheswara Rao G made changes -
          Link This issue is blocked by HDFS-3791 [ HDFS-3791 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-2815-branch-1.patch [ 12542794 ]
          Uma Maheswara Rao G made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 1.2.0, 2.0.0-alpha, 3.0.0 [ 12321657, 12320353, 12320356 ] 2.0.0-alpha, 1.2.0, 3.0.0 [ 12320353, 12321657, 12320356 ]
          Fix Version/s 1.1.1 [ 12321656 ]
          Resolution Fixed [ 1 ]
          Suresh Srinivas made changes -
          Fix Version/s 1.2.0 [ 12321657 ]
          Fix Version/s 1.1.1 [ 12321656 ]
          Target Version/s 1.2.0, 2.0.0-alpha, 3.0.0 [ 12321657, 12320353, 12320356 ] 2.0.0-alpha, 1.2.0, 3.0.0 [ 12320353, 12321657, 12320356 ]
          Matt Foley made changes -
          Target Version/s 1.2.0, 2.0.0-alpha, 3.0.0 [ 12321657, 12320353, 12320356 ] 2.0.0-alpha, 1.1.1, 3.0.0 [ 12320353, 12321656, 12320356 ]
          Matt Foley made changes -
          Fix Version/s 1.1.1 [ 12321656 ]
          Fix Version/s 1.2.0 [ 12321657 ]
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          sathish made changes -
          Link This issue is blocked by HDFS-5474 [ HDFS-5474 ]
          Vinayakumar B made changes -
          Link This issue is blocked by HDFS-5474 [ HDFS-5474 ]
          Vinayakumar B made changes -
          Link This issue is related to HDFS-5474 [ HDFS-5474 ]
          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320356 ]
          Allen Wittenauer made changes -
          Affects Version/s 2.0.0-alpha [ 12320353 ]
          Affects Version/s 0.24.0 [ 12317653 ]
          Allen Wittenauer made changes -
          Fix Version/s 0.23.2 [ 12319852 ]
          Fix Version/s 2.0.0-alpha [ 12320353 ]

            People

            • Assignee:
              Uma Maheswara Rao G
              Reporter:
              Uma Maheswara Rao G
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development