Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1636

If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      If dfs.name.dir points to an empty dir, namenode -format no longer requires confirmation.

      Description

      Right now, running namenode -format when dfs.name.dir is configured to a dir which exists but is empty still asks for confirmation. This is unnecessary since it isn't blowing away any real data.

      1. HDFS-1636.r3.diff
        3 kB
        Harsh J
      2. HDFS-1636.r2.diff
        2 kB
        Harsh J
      3. HDFS-1636.r1.diff
        0.9 kB
        Harsh J

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #704 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/704/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #704 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/704/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #685 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/685/)
        HDFS-1636. If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation. Contributed by Harsh J Chouraria.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130318
        Files :

        • /hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestAllowFormat.java
        • /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        • /hadoop/hdfs/trunk/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #685 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/685/ ) HDFS-1636 . If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation. Contributed by Harsh J Chouraria. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130318 Files : /hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestAllowFormat.java /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/hdfs/trunk/CHANGES.txt
        Todd Lipcon made changes -
        Release Note If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation. If dfs.name.dir points to an empty dir, namenode -format no longer requires confirmation.
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Hide
        Todd Lipcon added a comment -

        Committed to trunk. Thanks, Harsh!

        Show
        Todd Lipcon added a comment - Committed to trunk. Thanks, Harsh!
        Hide
        Todd Lipcon added a comment -

        +1. I manually tested this patch and it works great.

        Show
        Todd Lipcon added a comment - +1. I manually tested this patch and it works great.
        Hide
        Aaron T. Myers added a comment -

        Yup, this is unrelated. This was due to the commit of HDFS-1999 last night, and Todd fixed it this morning.

        Show
        Aaron T. Myers added a comment - Yup, this is unrelated. This was due to the commit of HDFS-1999 last night, and Todd fixed it this morning.
        Hide
        Harsh J added a comment -

        Failing system test framework compile appears to be cause of missing DFSConfigKeys. I'm positive I haven't touched that one here.

        [exec] -compile-fault-inject:
             [exec]      [echo] Start weaving aspects in place
             [exec]      [iajc] /grid/0/hudson/hudson-slave/workspace/PreCommit-HDFS-Build/trunk/src/test/system/java/org/apache/hadoop/hdfs/test/system/HDFSDaemonClient.java:38 [error] DFSConfigKeys cannot be resolved
             [exec]      [iajc] DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY);
             [exec]      [iajc] ^^^^
             [exec]      [iajc] /grid/0/hudson/hudson-slave/workspace/PreCommit-HDFS-Build/trunk/src/test/system/java/org/apache/hadoop/hdfs/test/system/HDFSDaemonClient.java:43 [error] DFSConfigKeys cannot be resolved
             [exec]      [iajc] DFSConfigKeys.DFS_NAMENODE_NAME_DIR_KEY)[0];
             [exec]      [iajc] ^^^^
             [exec]      [iajc] 
             [exec]      [iajc] 2 errors
        
        Show
        Harsh J added a comment - Failing system test framework compile appears to be cause of missing DFSConfigKeys. I'm positive I haven't touched that one here. [exec] -compile-fault-inject: [exec] [echo] Start weaving aspects in place [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/PreCommit-HDFS-Build/trunk/src/test/system/java/org/apache/hadoop/hdfs/test/system/HDFSDaemonClient.java:38 [error] DFSConfigKeys cannot be resolved [exec] [iajc] DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY); [exec] [iajc] ^^^^ [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/PreCommit-HDFS-Build/trunk/src/test/system/java/org/apache/hadoop/hdfs/test/system/HDFSDaemonClient.java:43 [error] DFSConfigKeys cannot be resolved [exec] [iajc] DFSConfigKeys.DFS_NAMENODE_NAME_DIR_KEY)[0]; [exec] [iajc] ^^^^ [exec] [iajc] [exec] [iajc] 2 errors
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12480541/HDFS-1636.r3.diff
        against trunk revision 1127823.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        -1 system test framework. The patch failed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/636//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/636//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/636//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480541/HDFS-1636.r3.diff against trunk revision 1127823. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/636//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/636//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/636//console This message is automatically generated.
        Harsh J made changes -
        Attachment HDFS-1636.r3.diff [ 12480541 ]
        Hide
        Harsh J added a comment -

        Todd, this new patch uses FileUtil.listFiles(…).length and ought to throw a proper IOException instead.

        Show
        Harsh J added a comment - Todd, this new patch uses FileUtil.listFiles(…).length and ought to throw a proper IOException instead.
        Hide
        Todd Lipcon added a comment -

        Hey Harsh. There's now an issue where the NN will throw an NPE if one of the directories exists but isn't accessible. eg I did "chmod 000 /my/namedir" and run namenode -format and got:

        11/05/25 09:43:32 ERROR namenode.NameNode: java.lang.NullPointerException
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1487)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1671)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1730)

        Consider looking at HADOOP-7322 for a utility method on its way into trunk to deal with this.

        Show
        Todd Lipcon added a comment - Hey Harsh. There's now an issue where the NN will throw an NPE if one of the directories exists but isn't accessible. eg I did "chmod 000 /my/namedir" and run namenode -format and got: 11/05/25 09:43:32 ERROR namenode.NameNode: java.lang.NullPointerException at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1487) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1671) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1730) Consider looking at HADOOP-7322 for a utility method on its way into trunk to deal with this.
        Hide
        Harsh J added a comment -

        Failing test looks unrelated.

        Show
        Harsh J added a comment - Failing test looks unrelated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12480416/HDFS-1636.r2.diff
        against trunk revision 1127390.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.TestDFSStorageStateRecovery

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/623//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/623//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/623//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480416/HDFS-1636.r2.diff against trunk revision 1127390. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSStorageStateRecovery +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/623//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/623//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/623//console This message is automatically generated.
        Harsh J made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Release Note If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation.
        Fix Version/s 0.23.0 [ 12315571 ]
        Harsh J made changes -
        Attachment HDFS-1636.r2.diff [ 12480416 ]
        Hide
        Harsh J added a comment -

        Added sort of a test case for this.

        Show
        Harsh J added a comment - Added sort of a test case for this.
        Hide
        Harsh J added a comment -

        I'd like some input on how to test if a case has called for CLI input or not?

        Or I could whip up a t-case where if it does not hang waiting for a CLI input would mean it passed (no real assertions?).

        Show
        Harsh J added a comment - I'd like some input on how to test if a case has called for CLI input or not? Or I could whip up a t-case where if it does not hang waiting for a CLI input would mean it passed (no real assertions?).
        Hide
        Aaron T. Myers added a comment -

        Hey Harsh, a test case is always strongly encouraged, especially when it's an easy test case to write like this would be.

        Show
        Aaron T. Myers added a comment - Hey Harsh, a test case is always strongly encouraged, especially when it's an easy test case to write like this would be.
        Harsh J made changes -
        Assignee Harsh J Chouraria [ qwertymaniac ]
        Harsh J made changes -
        Field Original Value New Value
        Attachment HDFS-1636.r1.diff [ 12478516 ]
        Hide
        Harsh J added a comment -

        Patch that adds this new condition.

        Would a test case also be required?

        Show
        Harsh J added a comment - Patch that adds this new condition. Would a test case also be required?
        Todd Lipcon created issue -

          People

          • Assignee:
            Harsh J
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development