Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15477 Enforce ordered snapshot deletion
  3. HDFS-15820

Ensure snapshot root trash provisioning happens only post safe mode exit

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Currently, on namenode startup, snapshot trash root provisioning starts as along with trash emptier service but namenode might not be out of safe mode by then. This can fail the snapshot trash dir creation thereby crashing the namenode. The idea here is to trigger snapshot trash provisioning only post safe mode exit.

      2021-02-04 11:23:47,323 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Error encountered requiring NN shutdown. Shutting down immediately.
      org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /upgrade/.Trash. Name node is in safe mode.
      The reported blocks 0 needs additional 1383 blocks to reach the threshold 0.9990 of total blocks 1385.
      The number of live datanodes 0 needs an additional 1 live datanodes to reach the minimum number 1.
      Safe mode will be turned off automatically once the thresholds have been reached. NamenodeHostName:quasar-brabeg-5.quasar-brabeg.root.hwx.site
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.newSafemodeException(FSNamesystem.java:1542)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1529)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3288)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAndProvisionSnapshotTrashRoots(FSNamesystem.java:8269)
              at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1939)
              at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:967)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:936)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1673)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1740)
      2021-02-04 11:23:47,334 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /upgrade/.Trash. Name node is in safe mode.
      

      Attachments

        Issue Links

          Activity

            People

              shashikant Shashikant Banerjee
              shashikant Shashikant Banerjee
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h