Hadoop Common
  1. Hadoop Common
  2. HADOOP-1980

'dfsadmin -safemode enter' should prevent the namenode from leaving safemode automatically after startup

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When debugging, I'd like to be able to intentionally keep the FS in a safemode. (For example, when looking at HADOOP-1978).
      Also, it'll be nice if the namenode can still update the webUI/report when it hits the dfs.safemode.threshold.pct.

      1. extendSafeMode.patch
        7 kB
        Konstantin Shvachko
      2. extendSafeMode-0-18.patch
        7 kB
        Konstantin Shvachko
      3. HADOOP-1980.patch
        8 kB
        Raghu Angadi
      4. HADOOP-1980.patch
        4 kB
        Raghu Angadi
      5. HADOOP-1980-branch-18.patch
        8 kB
        Raghu Angadi

        Issue Links

          Activity

          Koji Noguchi created issue -
          Koji Noguchi made changes -
          Field Original Value New Value
          Fix Version/s 0.16.0 [ 12312740 ]
          Allen Wittenauer made changes -
          Link This issue blocks HADOOP-2230 [ HADOOP-2230 ]
          Hide
          Allen Wittenauer added a comment -

          These two are sort of related. When HDFS is started up, it should be possible to select the mode that HDFS comes up in.

          Show
          Allen Wittenauer added a comment - These two are sort of related. When HDFS is started up, it should be possible to select the mode that HDFS comes up in.
          Nigel Daley made changes -
          Fix Version/s 0.16.0 [ 12312740 ]
          Raghu Angadi made changes -
          Assignee Raghu Angadi [ rangadi ]
          Hide
          Raghu Angadi added a comment -

          Suggested patch for trunk.

          If admin puts NN in safemode during startup, NN will stay in safemode even after block ratios are satisfied. It also prints block ratios for convenience (just like the default case where admin does not enter safemode manually).

          Show
          Raghu Angadi added a comment - Suggested patch for trunk. If admin puts NN in safemode during startup, NN will stay in safemode even after block ratios are satisfied. It also prints block ratios for convenience (just like the default case where admin does not enter safemode manually).
          Raghu Angadi made changes -
          Attachment HADOOP-1980.patch [ 12394206 ]
          Raghu Angadi made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12394206/HADOOP-1980.patch
          against trunk revision 719748.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3625/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3625/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3625/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3625/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12394206/HADOOP-1980.patch against trunk revision 719748. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3625/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3625/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3625/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3625/console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          The requirement here as I understand it is to make it possible to extend the safe mode indefinitely.
          First thing to do is just to start the name-node with a large extension or a >1 threshold. I guess this does not work if the name-node has already been started but the administrator needs to keep it in safe mode longer than the extension.
          Then why don't we just set really long extension in this case.
          Or even better provide an explicit option to extend safe mode from the admin command.

          hadoop dfsadmin -safemode extend
          

          This will be much simpler patch, which would satisfy all the requirements and will have clear user api.
          Otherwise it seems rather strange from a user point of view: in order to get infinite safe mode one should enter safe mode again although it is already on.

          Show
          Konstantin Shvachko added a comment - The requirement here as I understand it is to make it possible to extend the safe mode indefinitely. First thing to do is just to start the name-node with a large extension or a >1 threshold. I guess this does not work if the name-node has already been started but the administrator needs to keep it in safe mode longer than the extension. Then why don't we just set really long extension in this case. Or even better provide an explicit option to extend safe mode from the admin command. hadoop dfsadmin -safemode extend This will be much simpler patch, which would satisfy all the requirements and will have clear user api. Otherwise it seems rather strange from a user point of view: in order to get infinite safe mode one should enter safe mode again although it is already on.
          Hide
          Raghu Angadi added a comment -

          This does not preclude from setting a very large extension.

          This patch only brings consistent behavior to '-safemode enter'. Once you enter safemode manually, it makes good sense for it to require a manual command to leave. It should not matter weather it is already in safemode or not.

          Show
          Raghu Angadi added a comment - This does not preclude from setting a very large extension. This patch only brings consistent behavior to '-safemode enter'. Once you enter safemode manually, it makes good sense for it to require a manual command to leave. It should not matter weather it is already in safemode or not.
          Hide
          Raghu Angadi added a comment -

          Updated patch with a unit test.

          In some sense this is more of a bugfix rather than a new feature. It mainly aims to correct the the meaning of '-safemode enter'. Additionally normal information about % of blocks reported by datanodes is displayed to help administrators.

          Show
          Raghu Angadi added a comment - Updated patch with a unit test. In some sense this is more of a bugfix rather than a new feature. It mainly aims to correct the the meaning of '-safemode enter'. Additionally normal information about % of blocks reported by datanodes is displayed to help administrators.
          Raghu Angadi made changes -
          Attachment HADOOP-1980.patch [ 12396165 ]
          Raghu Angadi made changes -
          Attachment HADOOP-1980-branch-18.patch [ 12396245 ]
          Hide
          Konstantin Shvachko added a comment -

          Does not look like there is a big enthusiasm to make changes to the shell api.
          I simplified a bit Raghu's implementation of the feature.
          Renamed and tweaked the test so that it shutdowned the cluster in the final section if anything fails.

          Show
          Konstantin Shvachko added a comment - Does not look like there is a big enthusiasm to make changes to the shell api. I simplified a bit Raghu's implementation of the feature. Renamed and tweaked the test so that it shutdowned the cluster in the final section if anything fails.
          Konstantin Shvachko made changes -
          Attachment extendSafeMode.patch [ 12396458 ]
          Hide
          Boris Shkolnik added a comment -

          +1
          patch looks good

          Show
          Boris Shkolnik added a comment - +1 patch looks good
          Hide
          Konstantin Shvachko added a comment -

          This is patch for 0.18 branch.

          Show
          Konstantin Shvachko added a comment - This is patch for 0.18 branch.
          Konstantin Shvachko made changes -
          Attachment extendSafeMode-0-18.patch [ 12396533 ]
          Hide
          Konstantin Shvachko added a comment -

          I just committed this. Thank you Raghu.

          Show
          Konstantin Shvachko added a comment - I just committed this. Thank you Raghu.
          Konstantin Shvachko made changes -
          Assignee Raghu Angadi [ rangadi ] Konstantin Shvachko [ shv ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.18.3 [ 12313494 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]

            People

            • Assignee:
              Konstantin Shvachko
              Reporter:
              Koji Noguchi
            • Votes:
              2 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development