Hadoop Common
  1. Hadoop Common
  2. HADOOP-7325

hadoop command - do not accept class names starting with a hyphen

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: scripts
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If this is committed I will look at patches for hdfs and mapred.

      When teaching a good portion of the students in every single class execute:

      $ hadoop -fs ls /
      

      The -fs is passed directly to the JVM and the JVM fails to start:

      $ ./bin/hadoop -fs ls /
      Unrecognized option: -fs
      Could not create the Java virtual machine.
      

      Which is confusing and typically requires explanation. The attached patch improves that behavior:

      $ ./bin/hadoop -fs ls /
      Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
      

      The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:

      $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
      RunJar jarFile [mainClass] args...
      

      The hadoop command does not appear to advertise allowing JVM options before the classname.

      1. hadoop-illegal-class-name-2.patch
        0.6 kB
        Brock Noland
      2. hadoop-illegal-class-name-1.patch
        0.5 kB
        Brock Noland
      3. hadoop-illegal-class-name-0.patch
        0.7 kB
        Brock Noland

        Issue Links

          Activity

          Brock Noland created issue -
          Hide
          Brock Noland added a comment -

          Attached implements requested improvement.

          Show
          Brock Noland added a comment - Attached implements requested improvement.
          Brock Noland made changes -
          Field Original Value New Value
          Attachment hadoop-illegal-class-name-0.patch [ 12480193 ]
          Todd Lipcon made changes -
          Assignee Brock Noland [ brocknoland ]
          Hide
          Aaron T. Myers added a comment -

          Hitting 'submit patch' so Hudson will run the pre-commit hooks.

          Show
          Aaron T. Myers added a comment - Hitting 'submit patch' so Hudson will run the pre-commit hooks.
          Aaron T. Myers made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480193/hadoop-illegal-class-name-0.patch
          against trunk revision 1128385.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/535//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/535//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/535//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480193/hadoop-illegal-class-name-0.patch against trunk revision 1128385. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/535//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/535//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/535//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          nits:

          • I don't think you need the comments explaining bash [[ vs [
          • if you do want to keep them, there's a typo: bra*c*ket is missing its 'c'.
          Show
          Todd Lipcon added a comment - nits: I don't think you need the comments explaining bash [[ vs [ if you do want to keep them, there's a typo: bra*c*ket is missing its 'c'.
          Hide
          Brock Noland added a comment -

          nit's resolved.

          Show
          Brock Noland added a comment - nit's resolved.
          Brock Noland made changes -
          Attachment hadoop-illegal-class-name-1.patch [ 12481405 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481405/hadoop-illegal-class-name-1.patch
          against trunk revision 1130833.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/572//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/572//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/572//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481405/hadoop-illegal-class-name-1.patch against trunk revision 1130833. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/572//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/572//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/572//console This message is automatically generated.
          Hide
          Brock Noland added a comment -

          Slightly too agressive on comment deletion. I think the first comment was useful. Updated.

          Show
          Brock Noland added a comment - Slightly too agressive on comment deletion. I think the first comment was useful. Updated.
          Brock Noland made changes -
          Attachment hadoop-illegal-class-name-2.patch [ 12481408 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481408/hadoop-illegal-class-name-2.patch
          against trunk revision 1130833.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/574//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/574//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/574//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481408/hadoop-illegal-class-name-2.patch against trunk revision 1130833. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/574//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/574//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/574//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          +1

          Show
          Todd Lipcon added a comment - +1
          Todd Lipcon made changes -
          Hadoop Flags [Reviewed]
          Hide
          Todd Lipcon added a comment -

          Committed to trunk and 0.22. Thanks, Brock! Please do file patches for bin/hdfs and bin/mapred in those projects as well.

          Show
          Todd Lipcon added a comment - Committed to trunk and 0.22. Thanks, Brock! Please do file patches for bin/hdfs and bin/mapred in those projects as well.
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.22.0 [ 12314296 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #643 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/643/)
          HADOOP-7325. The hadoop command should not accept class names starting with a hyphen. Contributed by Brock Noland.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132769
          Files :

          • /hadoop/common/trunk/bin/hadoop
          • /hadoop/common/trunk/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #643 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/643/ ) HADOOP-7325 . The hadoop command should not accept class names starting with a hyphen. Contributed by Brock Noland. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132769 Files : /hadoop/common/trunk/bin/hadoop /hadoop/common/trunk/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-22-branch #63 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/63/)
          HADOOP-7325. The hadoop command should not accept class names starting with a hyphen. Contributed by Brock Noland.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132768
          Files :

          • /hadoop/common/branches/branch-0.22/bin/hadoop
          • /hadoop/common/branches/branch-0.22/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Common-22-branch #63 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/63/ ) HADOOP-7325 . The hadoop command should not accept class names starting with a hyphen. Contributed by Brock Noland. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132768 Files : /hadoop/common/branches/branch-0.22/bin/hadoop /hadoop/common/branches/branch-0.22/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #712 (See https://builds.apache.org/job/Hadoop-Common-trunk/712/)
          HADOOP-7325. The hadoop command should not accept class names starting with a hyphen. Contributed by Brock Noland.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132769
          Files :

          • /hadoop/common/trunk/bin/hadoop
          • /hadoop/common/trunk/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #712 (See https://builds.apache.org/job/Hadoop-Common-trunk/712/ ) HADOOP-7325 . The hadoop command should not accept class names starting with a hyphen. Contributed by Brock Noland. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132769 Files : /hadoop/common/trunk/bin/hadoop /hadoop/common/trunk/CHANGES.txt
          Hide
          Brock Noland added a comment -

          Looking at the hdfs and mapred commands, they do not seem to allow the "command" to be a direct classname and also have unknown command handling built in, so I do not think they need to be updated. If someone disagrees, please share!

          elif [ "$COMMAND" = "groups" ] ; then
            CLASS=org.apache.hadoop.hdfs.tools.GetGroups
          else
            echo $COMMAND - invalid command
            print_usage
            exit
          
          Show
          Brock Noland added a comment - Looking at the hdfs and mapred commands, they do not seem to allow the "command" to be a direct classname and also have unknown command handling built in, so I do not think they need to be updated. If someone disagrees, please share! elif [ "$COMMAND" = "groups" ] ; then CLASS=org.apache.hadoop.hdfs.tools.GetGroups else echo $COMMAND - invalid command print_usage exit
          Hide
          Todd Lipcon added a comment -

          Sounds right to me. Thanks for looking.

          Show
          Todd Lipcon added a comment - Sounds right to me. Thanks for looking.
          Eli Collins made changes -
          Assignee Brock Noland [ brocknoland ] Eli Collins [ eli ]
          Eli Collins made changes -
          Assignee Eli Collins [ eli ] Brock Noland [ brocknoland ]
          Eli Collins made changes -
          Link This issue incorporates HADOOP-7491 [ HADOOP-7491 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          3d 17h 28m 1 Aaron T. Myers 27/May/11 19:49
          Patch Available Patch Available Resolved Resolved
          10d 2h 9m 1 Todd Lipcon 06/Jun/11 21:58

            People

            • Assignee:
              Brock Noland
              Reporter:
              Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development