Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2580

NameNode#main(...) can make use of GenericOptionsParser.

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 3.0.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      DataNode supports passing generic opts when calling via hdfs datanode. NameNode can support the same thing as well, but doesn't right now.

        Activity

        Hide
        Harsh J added a comment -

        Or perhaps we restructure NN/DN to use the Tool interface?

        Show
        Harsh J added a comment - Or perhaps we restructure NN/DN to use the Tool interface?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12504565/HDFS-2580.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.fs.viewfs.TestViewFsHdfs

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1599//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1599//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504565/HDFS-2580.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.viewfs.TestViewFsHdfs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1599//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1599//console This message is automatically generated.
        Hide
        Harsh J added a comment -

        Resubmitting for tests.

        I don't see an elegant way to use Tool interface, given the createNamenode(…) static call required to initialize 'this'. This should suffice.

        Show
        Harsh J added a comment - Resubmitting for tests. I don't see an elegant way to use Tool interface, given the createNamenode(…) static call required to initialize 'this'. This should suffice.
        Hide
        Harsh J added a comment -

        Any comments on this addition? Really does help devs when they start NN via CLI directly.

        Show
        Harsh J added a comment - Any comments on this addition? Really does help devs when they start NN via CLI directly.
        Hide
        Eli Collins added a comment -

        Is the motivation so you can use -D? None of the other generic options seem like they apply to the DN or NN.

        Show
        Eli Collins added a comment - Is the motivation so you can use -D? None of the other generic options seem like they apply to the DN or NN.
        Hide
        Harsh J added a comment -

        Eli,

        Yes the motivation is for allowing -D params, to help me switch dfs.name.dir and other such properties rather than having to create XML config files for each instance I wish to start.

        Show
        Harsh J added a comment - Eli, Yes the motivation is for allowing -D params, to help me switch dfs.name.dir and other such properties rather than having to create XML config files for each instance I wish to start.
        Hide
        Harsh J added a comment -

        (Btw -fs applies too, afaict, and thats what my main motivation was, aside of dfs.namenode.name.dir and such)

        Show
        Harsh J added a comment - (Btw -fs applies too, afaict, and thats what my main motivation was, aside of dfs.namenode.name.dir and such)
        Hide
        Harsh J added a comment -

        Is this worthy? We can also have tests to ensure no daemons regress away from not using Tool/GOP parsers in their main. If agreed, I shall post up some work that does this coverage.

        Show
        Harsh J added a comment - Is this worthy? We can also have tests to ensure no daemons regress away from not using Tool/GOP parsers in their main. If agreed, I shall post up some work that does this coverage.
        Hide
        Eli Collins added a comment -

        Hey Harsh,

        How does -fs apply? I'm not sure adding support for the tools interface is the right way to accomplish this. Can you tell me more about what you're trying to achieve, not having to edit conf files?

        Show
        Eli Collins added a comment - Hey Harsh, How does -fs apply? I'm not sure adding support for the tools interface is the right way to accomplish this. Can you tell me more about what you're trying to achieve, not having to edit conf files?
        Hide
        Harsh J added a comment -

        Yes I wished to avoid editing conf files when bringing up NNs locally for testing with some tweaked params. The use-case I had was to start NN via "hadoop namenode -Dopts=values". Most of the daemons have a GOP (if not Tool, which is fine too), except for NameNode.

        The -fs helps set the port.

        Show
        Harsh J added a comment - Yes I wished to avoid editing conf files when bringing up NNs locally for testing with some tweaked params. The use-case I had was to start NN via "hadoop namenode -Dopts=values". Most of the daemons have a GOP (if not Tool, which is fine too), except for NameNode. The -fs helps set the port.
        Hide
        Eli Collins added a comment -

        Makes sense. +1

        Show
        Eli Collins added a comment - Makes sense. +1
        Hide
        Harsh J added a comment -

        Thanks Eli. Rebased and committed to trunk as r1379828.

        Show
        Harsh J added a comment - Thanks Eli. Rebased and committed to trunk as r1379828.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2671 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2671/)
        HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828)

        Result = SUCCESS
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2671 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2671/ ) HDFS-2580 . NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2734 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2734/)
        HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828)

        Result = SUCCESS
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2734 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2734/ ) HDFS-2580 . NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2696 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2696/)
        HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828)

        Result = FAILURE
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2696 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2696/ ) HDFS-2580 . NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1153 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1153/)
        HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828)

        Result = FAILURE
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1153 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1153/ ) HDFS-2580 . NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1184 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1184/)
        HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828)

        Result = FAILURE
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1184 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1184/ ) HDFS-2580 . NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java

          People

          • Assignee:
            Harsh J
            Reporter:
            Harsh J
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development