Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4021

Misleading error message when resources are low on the NameNode

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 2.0.3-alpha
    • 2.0.3-alpha
    • namenode

    Description

      When resources are low on the namenode, it enters SafeMode with a message like this:

      14:00:10,666  WARN NameNodeResourceChecker:89 - Space available on
      volume '/dev/hda1' is 104357888, which is below the configured
      reserved amount 104857600
      14:00:10,670  WARN FSNamesystem:3190 - NameNode low on available disk
      space. Entering safe mode.
      14:00:10,670  INFO StateChange:3836 - STATE* Safe mode is ON.
      Resources are low on NN. Safe mode must be turned off manually.
      

      However, turning off safe mode manually has no effect, since it immediately puts itself into Safe Mode again with a log message like this:

       14:00:10,666  WARN NameNodeResourceChecker:89 - Space available on
      volume '/dev/hda1' is 104357888, which is below the configured
      reserved amount 104857600
      14:00:10,670  WARN FSNamesystem:3190 - NameNode low on available disk
      space. Entering safe mode.
      14:00:10,670  INFO StateChange:3836 - STATE* Safe mode is ON.
      Resources are low on NN. Safe mode must be turned off manually.
      

      From the shell, it looks like this:

       [cmccabe@vm1 h]$ ./bin/hdfs dfsadmin -safemode get
      Safe mode is ON
      [cmccabe@vm1 h]$ ./bin/hdfs dfsadmin -safemode leave
      Safe mode is OFF
      [cmccabe@vm1 h]$ ./bin/hdfs dfsadmin -safemode get
      Safe mode is ON
      

      It seems like we should change the message about turning off safe mode manually, if turning off safe mode manually does not actually work for the case where resources are low. Probably we need to explain that safe mode should be turned off manually after adding more resources. As it is, the error message seems misleading.

      Attachments

        1. HDFS-4021.patch
          1 kB
          Christopher Conner
        2. HDFS-4021.patch
          1 kB
          Christopher Conner

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cconner Christopher Conner
            cmccabe Colin McCabe
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment