Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10903

Enhance hadoop classpath command to expand wildcards or write classpath into jar manifest.

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: scripts, util
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      The "hadoop classpath" command has been enhanced to support options for automatic expansion of wildcards in classpath elements and writing the classpath to a jar file manifest. These options make it easier to construct a correct classpath for libhdfs applications.
      Show
      The "hadoop classpath" command has been enhanced to support options for automatic expansion of wildcards in classpath elements and writing the classpath to a jar file manifest. These options make it easier to construct a correct classpath for libhdfs applications.

      Description

      The "hadoop classpath" shell command currently prints the classpath variable established by the shell scripts and then exits. A few enhancements to this command would be desirable to support a few other use cases.

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          Typically, the output of "hadoop classpath" includes wildcards, because we take advantage of wildcard syntax to simplify our classpath. Two enhancements would be beneficial:

          1. An option to print the classpath after wildcard expansion. This is helpful for use in environments that need the classpath but don't support the wildcard syntax. One example is libhdfs, where a common user error is incorrectly setting the CLASSPATH environment variable with wildcards.
          2. An option to write the classpath into a jar file manifest. This helps work around command line length limitations with our very long classpaths. For example, Windows command prompt is limited to 8191 characters. We've already used the technique of packing the classpath into a jar manifest successfully in the YARN NodeManager.

          The default behavior can still be to print the classpath calculated by the shell scripts and then exit immediately. This is still valuable behavior for getting a glimpse of the classpath before JVM launch, just in case the scripts are doing something completely unexpected.

          Show
          cnauroth Chris Nauroth added a comment - Typically, the output of "hadoop classpath" includes wildcards, because we take advantage of wildcard syntax to simplify our classpath. Two enhancements would be beneficial: An option to print the classpath after wildcard expansion. This is helpful for use in environments that need the classpath but don't support the wildcard syntax. One example is libhdfs, where a common user error is incorrectly setting the CLASSPATH environment variable with wildcards. An option to write the classpath into a jar file manifest. This helps work around command line length limitations with our very long classpaths. For example, Windows command prompt is limited to 8191 characters. We've already used the technique of packing the classpath into a jar manifest successfully in the YARN NodeManager. The default behavior can still be to print the classpath calculated by the shell scripts and then exit immediately. This is still valuable behavior for getting a glimpse of the classpath before JVM launch, just in case the scripts are doing something completely unexpected.
          Hide
          cnauroth Chris Nauroth added a comment -

          Here is the patch implementing what I described in my last comment. In addition to the new test suite included in the patch, I've manually tested using these options to build the CLASSPATH environment variable for a libhdfs application running on both Linux and Windows.

          Show
          cnauroth Chris Nauroth added a comment - Here is the patch implementing what I described in my last comment. In addition to the new test suite included in the patch, I've manually tested using these options to build the CLASSPATH environment variable for a libhdfs application running on both Linux and Windows.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12658678/HADOOP-10903.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.ipc.TestRPCCallBenchmark
          org.apache.hadoop.net.TestNetUtils
          org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
          org.apache.hadoop.security.TestSecurityUtil
          org.apache.hadoop.ipc.TestRPC
          org.apache.hadoop.security.TestDoAsEffectiveUser
          org.apache.hadoop.conf.TestConfiguration
          org.apache.hadoop.ipc.TestIPC

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4391//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4391//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658678/HADOOP-10903.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ipc.TestRPCCallBenchmark org.apache.hadoop.net.TestNetUtils org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl org.apache.hadoop.security.TestSecurityUtil org.apache.hadoop.ipc.TestRPC org.apache.hadoop.security.TestDoAsEffectiveUser org.apache.hadoop.conf.TestConfiguration org.apache.hadoop.ipc.TestIPC +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4391//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4391//console This message is automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          The failures are unrelated. I see a lot of bizarre unknown host errors in the log, which can't possibly be connected to this patch.

          Show
          cnauroth Chris Nauroth added a comment - -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: The failures are unrelated. I see a lot of bizarre unknown host errors in the log, which can't possibly be connected to this patch.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Chris,

              if (cf.getOpt("-glob")) {
                // The classpath returned from the property has been globbed already.
                System.out.println(classPath);
          

          I am probably missing something obvious, who performs the globbing if '-glob' is specified?

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Chris, if (cf.getOpt( "-glob" )) { // The classpath returned from the property has been globbed already. System .out.println(classPath); I am probably missing something obvious, who performs the globbing if '-glob' is specified?
          Hide
          cnauroth Chris Nauroth added a comment -

          I am probably missing something obvious, who performs the globbing if '-glob' is specified?

          java does this for us behind the scenes. When we do System.getProperty("java.class.path"), java has already expanded all of the wildcards in the input class path. This property already contains the "real" classpath with the path to every individual jar or directory, so we don't need to glob explicitly.

          Show
          cnauroth Chris Nauroth added a comment - I am probably missing something obvious, who performs the globbing if '-glob' is specified? java does this for us behind the scenes. When we do System.getProperty("java.class.path") , java has already expanded all of the wildcards in the input class path. This property already contains the "real" classpath with the path to every individual jar or directory, so we don't need to glob explicitly.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks Chris. +1 for the patch. Verified that globbing works as expected.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks Chris. +1 for the patch. Verified that globbing works as expected.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6003 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6003/)
          HADOOP-10903. Enhance hadoop classpath command to expand wildcards or write classpath into jar manifest. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1615386)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Classpath.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestClasspath.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6003 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6003/ ) HADOOP-10903 . Enhance hadoop classpath command to expand wildcards or write classpath into jar manifest. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1615386 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Classpath.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestClasspath.java
          Hide
          cnauroth Chris Nauroth added a comment -

          Thanks for the code review, Arpit. I committed it to trunk and branch-2.

          Show
          cnauroth Chris Nauroth added a comment - Thanks for the code review, Arpit. I committed it to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #632 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/632/)
          HADOOP-10903. Enhance hadoop classpath command to expand wildcards or write classpath into jar manifest. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1615386)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Classpath.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestClasspath.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #632 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/632/ ) HADOOP-10903 . Enhance hadoop classpath command to expand wildcards or write classpath into jar manifest. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1615386 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Classpath.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestClasspath.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1826 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1826/)
          HADOOP-10903. Enhance hadoop classpath command to expand wildcards or write classpath into jar manifest. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1615386)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Classpath.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestClasspath.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1826 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1826/ ) HADOOP-10903 . Enhance hadoop classpath command to expand wildcards or write classpath into jar manifest. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1615386 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Classpath.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestClasspath.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1851 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1851/)
          HADOOP-10903. Enhance hadoop classpath command to expand wildcards or write classpath into jar manifest. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1615386)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Classpath.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestClasspath.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1851 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1851/ ) HADOOP-10903 . Enhance hadoop classpath command to expand wildcards or write classpath into jar manifest. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1615386 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Classpath.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestClasspath.java
          Hide
          raviprak Ravi Prakash added a comment -

          It seems like wildcards aren't truly supported. For e.g. if I want to include /pathA/pathB/pathC/someJar1-v1.2.3.jar and added /pathA/pathB/pathC/someJar1-.jar to HADOOP_CLASSPATH, the output of hadoop classpath -glob still contains /pathA/pathB/pathC/someJar1.jar . To include what I wanted to include, I have to add /pathA/pathB/pathC/* to HADOOP_CLASSPATH. Could someone please confirm?

          Show
          raviprak Ravi Prakash added a comment - It seems like wildcards aren't truly supported. For e.g. if I want to include /pathA/pathB/pathC/someJar1-v1.2.3.jar and added /pathA/pathB/pathC/someJar1- .jar to HADOOP_CLASSPATH, the output of hadoop classpath - glob still contains /pathA/pathB/pathC/someJar1 .jar . To include what I wanted to include, I have to add /pathA/pathB/pathC/* to HADOOP_CLASSPATH. Could someone please confirm?
          Hide
          raviprak Ravi Prakash added a comment -

          It seems like wildcards aren't truly supported. For e.g. if I want to include /pathA/pathB/pathC/someJar1-v1.2.3.jar and added /pathA/pathB/pathC/someJar1-*.jar to HADOOP_CLASSPATH, the output of hadoop classpath -glob still contains /pathA/pathB/pathC/someJar1-*.jar . To include what I wanted to include, I have to add /pathA/pathB/pathC/* to HADOOP_CLASSPATH. Could someone please confirm? Ofcourse this pulls in all the jars in pathC
          Sorry for the bad formatting in the earlier comment

          Show
          raviprak Ravi Prakash added a comment - It seems like wildcards aren't truly supported. For e.g. if I want to include /pathA/pathB/pathC/someJar1-v1.2.3.jar and added /pathA/pathB/pathC/someJar1-*.jar to HADOOP_CLASSPATH , the output of hadoop classpath -glob still contains /pathA/pathB/pathC/someJar1-*.jar . To include what I wanted to include, I have to add /pathA/pathB/pathC/* to HADOOP_CLASSPATH . Could someone please confirm? Ofcourse this pulls in all the jars in pathC Sorry for the bad formatting in the earlier comment
          Hide
          raviprak Ravi Prakash added a comment -

          It seems like we are depending on JAVA to resolve the wildcards in the classpath. The Java documentation explains the behavior I am seeing.

          Show
          raviprak Ravi Prakash added a comment - It seems like we are depending on JAVA to resolve the wildcards in the classpath. The Java documentation explains the behavior I am seeing.

            People

            • Assignee:
              cnauroth Chris Nauroth
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development