Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2580

NameNode#main(...) can make use of GenericOptionsParser.

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      DataNode supports passing generic opts when calling via hdfs datanode. NameNode can support the same thing as well, but doesn't right now.

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
        Hide
        shv Konstantin Shvachko added a comment -

        Committed this to branch-2, branch-2.8, branch-2.7, branch-2.6.

        Show
        shv Konstantin Shvachko added a comment - Committed this to branch-2, branch-2.8, branch-2.7, branch-2.6.
        Hide
        shv Konstantin Shvachko added a comment -

        This is an old patch, but never made it to any releases. I'd propose to commit it down all the way up to branch-2.6.

        Show
        shv Konstantin Shvachko added a comment - This is an old patch, but never made it to any releases. I'd propose to commit it down all the way up to branch-2.6.
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1184 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1184/)
        HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828)

        Result = FAILURE
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1184 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1184/ ) HDFS-2580 . NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1153 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1153/)
        HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828)

        Result = FAILURE
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1153 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1153/ ) HDFS-2580 . NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2696 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2696/)
        HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828)

        Result = FAILURE
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2696 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2696/ ) HDFS-2580 . NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2734 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2734/)
        HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828)

        Result = SUCCESS
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2734 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2734/ ) HDFS-2580 . NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2671 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2671/)
        HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828)

        Result = SUCCESS
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2671 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2671/ ) HDFS-2580 . NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) (Revision 1379828) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379828 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Hide
        qwertymaniac Harsh J added a comment -

        Thanks Eli. Rebased and committed to trunk as r1379828.

        Show
        qwertymaniac Harsh J added a comment - Thanks Eli. Rebased and committed to trunk as r1379828.
        Hide
        eli Eli Collins added a comment -

        Makes sense. +1

        Show
        eli Eli Collins added a comment - Makes sense. +1
        Hide
        qwertymaniac Harsh J added a comment -

        Yes I wished to avoid editing conf files when bringing up NNs locally for testing with some tweaked params. The use-case I had was to start NN via "hadoop namenode -Dopts=values". Most of the daemons have a GOP (if not Tool, which is fine too), except for NameNode.

        The -fs helps set the port.

        Show
        qwertymaniac Harsh J added a comment - Yes I wished to avoid editing conf files when bringing up NNs locally for testing with some tweaked params. The use-case I had was to start NN via "hadoop namenode -Dopts=values". Most of the daemons have a GOP (if not Tool, which is fine too), except for NameNode. The -fs helps set the port.
        Hide
        eli Eli Collins added a comment -

        Hey Harsh,

        How does -fs apply? I'm not sure adding support for the tools interface is the right way to accomplish this. Can you tell me more about what you're trying to achieve, not having to edit conf files?

        Show
        eli Eli Collins added a comment - Hey Harsh, How does -fs apply? I'm not sure adding support for the tools interface is the right way to accomplish this. Can you tell me more about what you're trying to achieve, not having to edit conf files?
        Hide
        qwertymaniac Harsh J added a comment -

        Is this worthy? We can also have tests to ensure no daemons regress away from not using Tool/GOP parsers in their main. If agreed, I shall post up some work that does this coverage.

        Show
        qwertymaniac Harsh J added a comment - Is this worthy? We can also have tests to ensure no daemons regress away from not using Tool/GOP parsers in their main. If agreed, I shall post up some work that does this coverage.
        Hide
        qwertymaniac Harsh J added a comment -

        (Btw -fs applies too, afaict, and thats what my main motivation was, aside of dfs.namenode.name.dir and such)

        Show
        qwertymaniac Harsh J added a comment - (Btw -fs applies too, afaict, and thats what my main motivation was, aside of dfs.namenode.name.dir and such)
        Hide
        qwertymaniac Harsh J added a comment -

        Eli,

        Yes the motivation is for allowing -D params, to help me switch dfs.name.dir and other such properties rather than having to create XML config files for each instance I wish to start.

        Show
        qwertymaniac Harsh J added a comment - Eli, Yes the motivation is for allowing -D params, to help me switch dfs.name.dir and other such properties rather than having to create XML config files for each instance I wish to start.
        Hide
        eli2 Eli Collins added a comment -

        Is the motivation so you can use -D? None of the other generic options seem like they apply to the DN or NN.

        Show
        eli2 Eli Collins added a comment - Is the motivation so you can use -D? None of the other generic options seem like they apply to the DN or NN.
        Hide
        qwertymaniac Harsh J added a comment -

        Any comments on this addition? Really does help devs when they start NN via CLI directly.

        Show
        qwertymaniac Harsh J added a comment - Any comments on this addition? Really does help devs when they start NN via CLI directly.
        Hide
        qwertymaniac Harsh J added a comment -

        Resubmitting for tests.

        I don't see an elegant way to use Tool interface, given the createNamenode(…) static call required to initialize 'this'. This should suffice.

        Show
        qwertymaniac Harsh J added a comment - Resubmitting for tests. I don't see an elegant way to use Tool interface, given the createNamenode(…) static call required to initialize 'this'. This should suffice.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12504565/HDFS-2580.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.fs.viewfs.TestViewFsHdfs

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1599//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1599//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504565/HDFS-2580.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.viewfs.TestViewFsHdfs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1599//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1599//console This message is automatically generated.
        Hide
        qwertymaniac Harsh J added a comment -

        Or perhaps we restructure NN/DN to use the Tool interface?

        Show
        qwertymaniac Harsh J added a comment - Or perhaps we restructure NN/DN to use the Tool interface?

          People

          • Assignee:
            qwertymaniac Harsh J
            Reporter:
            qwertymaniac Harsh J
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development