Hadoop Common
  1. Hadoop Common
  2. HADOOP-5961

DataNode should understand generic hadoop options

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      hdfs datanode

      Description

      DataNode should use ToolRunner to parse generic hadoop options (like -Dconfig.var=value). Apart from being more consistent with rest of the hadoop programs, this is necessary to write a simple script to run multiple datanodes on the same node.

      I had proposed a simple shell script to running multiple datanodes on a the user mailing list http://www.nabble.com/Contributing-to-hadoop-td22198897.html, but it requires that DataNode parses -D option.

      I will submit a patch and a shell script that works.

      1. run-datanode.sh
        0.8 kB
        Raghu Angadi
      2. run-datanode.sh
        0.8 kB
        Raghu Angadi
      3. HADOOP-5961.patch
        1 kB
        Raghu Angadi
      4. HADOOP-5961.patch
        1 kB
        Raghu Angadi

        Issue Links

          Activity

          Hide
          Raghu Angadi added a comment -

          run-datanode.sh : an example script to start multiple datanodes on one machine.

          Show
          Raghu Angadi added a comment - run-datanode.sh : an example script to start multiple datanodes on one machine.
          Hide
          Raghu Angadi added a comment -

          Patch attached. DataNode does not need to implement 'Tool'. It can just use GenericOptionsParser directly.

          Show
          Raghu Angadi added a comment - Patch attached. DataNode does not need to implement 'Tool'. It can just use GenericOptionsParser directly.
          Hide
          Raghu Angadi added a comment -

          This does not need unit tests.

          Show
          Raghu Angadi added a comment - This does not need unit tests.
          Hide
          Jakob Homan added a comment -

          +1 on the patch. Works as advertised. Is there any reason not to include the script and integrate it with the other shell scripts? The script works fine, but would need some validation (can only open 10 datanodes as it stands), header file, etc. This would make it easy for other developers to use without having to track down this JIRA months or years from now.

          Show
          Jakob Homan added a comment - +1 on the patch. Works as advertised. Is there any reason not to include the script and integrate it with the other shell scripts? The script works fine, but would need some validation (can only open 10 datanodes as it stands), header file, etc. This would make it easy for other developers to use without having to track down this JIRA months or years from now.
          Hide
          Raghu Angadi added a comment -

          Thanks Jakob.

          The current script is extremely basic, but gets the job done. It is more like command line alias. Still I spend more time the script than the actual code change in DataNode. It would take some time to write a good script and for associated help and support. We could consider later that if there is more interest.

          Show
          Raghu Angadi added a comment - Thanks Jakob. The current script is extremely basic, but gets the job done. It is more like command line alias. Still I spend more time the script than the actual code change in DataNode. It would take some time to write a good script and for associated help and support. We could consider later that if there is more interest.
          Hide
          Raghu Angadi added a comment -

          run-datanode.sh : Minor typo fixes

          Show
          Raghu Angadi added a comment - run-datanode.sh : Minor typo fixes
          Hide
          Raghu Angadi added a comment -

          Reattach the patch so that Hudson picks up the right one.

          Show
          Raghu Angadi added a comment - Reattach the patch so that Hudson picks up the right one.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12409907/HADOOP-5961.patch
          against trunk revision 781816.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/464/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/464/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/464/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/464/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12409907/HADOOP-5961.patch against trunk revision 781816. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/464/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/464/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/464/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/464/console This message is automatically generated.
          Hide
          Raghu Angadi added a comment -

          I just committed this.

          Show
          Raghu Angadi added a comment - I just committed this.
          Hide
          Raghu Angadi added a comment -

          Fix 'Fix Version'. Thanks Koji.

          Show
          Raghu Angadi added a comment - Fix 'Fix Version'. Thanks Koji.

            People

            • Assignee:
              Raghu Angadi
              Reporter:
              Raghu Angadi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development