Hadoop Common
  1. Hadoop Common
  2. HADOOP-9488

FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 2.1.0-beta
    • Component/s: fs
    • Labels:
      None
    • Target Version/s:

      Description

      FileUtil#createJarWithClassPath always uses System#getEnv for substitution of environment variables in the classpath bundled into the jar manifest. YARN launches container processes with a different set of environment variables, so the method needs to support providing environment variables different from the current process.

      1. HADOOP-9488.consolidated.2.patch
        17 kB
        Chris Nauroth
      2. HADOOP-9488.consolidated.1.patch
        17 kB
        Chris Nauroth
      3. HADOOP-9488.1.patch
        5 kB
        Chris Nauroth

        Issue Links

          Activity

          Chris Nauroth created issue -
          Hide
          Chris Nauroth added a comment -

          This jira tracks a change that originated in MAPREDUCE-4987. The Hadoop Common portion of that patch will be attached here.

          Show
          Chris Nauroth added a comment - This jira tracks a change that originated in MAPREDUCE-4987 . The Hadoop Common portion of that patch will be attached here.
          Hide
          Chris Nauroth added a comment -

          Attaching patch containing just the Hadoop Common portion of the changes. This code has already been reviewed as part of MAPREDUCE-4987.

          Show
          Chris Nauroth added a comment - Attaching patch containing just the Hadoop Common portion of the changes. This code has already been reviewed as part of MAPREDUCE-4987 .
          Chris Nauroth made changes -
          Field Original Value New Value
          Attachment HADOOP-9488.1.patch [ 12579571 ]
          Chris Nauroth made changes -
          Link This issue relates to YARN-593 [ YARN-593 ]
          Chris Nauroth made changes -
          Link This issue relates to MAPREDUCE-4987 [ MAPREDUCE-4987 ]
          Chris Nauroth made changes -
          Link This issue is part of HADOOP-8562 [ HADOOP-8562 ]
          Hide
          Chris Nauroth added a comment -

          Attaching consolidated patch of all changes previously attached to MAPREDUCE-4987, so that we can get a Jenkins run here.

          Show
          Chris Nauroth added a comment - Attaching consolidated patch of all changes previously attached to MAPREDUCE-4987 , so that we can get a Jenkins run here.
          Chris Nauroth made changes -
          Attachment HADOOP-9488.1.patch [ 12579574 ]
          Chris Nauroth made changes -
          Attachment HADOOP-9488.1.patch [ 12579574 ]
          Chris Nauroth made changes -
          Attachment HADOOP-9488.consolidated.1.patch [ 12579576 ]
          Chris Nauroth made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12579576/HADOOP-9488.consolidated.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2460//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2460//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579576/HADOOP-9488.consolidated.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2460//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2460//console This message is automatically generated.
          Hide
          Chris Nauroth added a comment -

          Updating consolidated patch to check isDirectory just once in the outer loop. (See comments on YARN-593.)

          Show
          Chris Nauroth added a comment - Updating consolidated patch to check isDirectory just once in the outer loop. (See comments on YARN-593 .)
          Chris Nauroth made changes -
          Attachment HADOOP-9488.consolidated.2.patch [ 12579603 ]
          Hide
          Bikas Saha added a comment -

          +1. Committed to trunk.

          Show
          Bikas Saha added a comment - +1. Committed to trunk.
          Bikas Saha made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3636 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3636/)
          HADOOP-9488. FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996)

          Result = SUCCESS
          bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3636 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3636/ ) HADOOP-9488 . FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12579603/HADOOP-9488.consolidated.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2461//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2461//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579603/HADOOP-9488.consolidated.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2461//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2461//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #189 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/189/)
          HADOOP-9488. FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996)

          Result = SUCCESS
          bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #189 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/189/ ) HADOOP-9488 . FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1378 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1378/)
          HADOOP-9488. FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996)

          Result = FAILURE
          bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1378 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1378/ ) HADOOP-9488 . FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996) Result = FAILURE bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1405 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1405/)
          HADOOP-9488. FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996)

          Result = SUCCESS
          bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1405 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1405/ ) HADOOP-9488 . FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
          Hide
          Suresh Srinivas added a comment -

          I merged the patch to branch-2.

          Show
          Suresh Srinivas added a comment - I merged the patch to branch-2.
          Suresh Srinivas made changes -
          Fix Version/s 2.0.5-beta [ 12324030 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          15m 26s 1 Chris Nauroth 19/Apr/13 18:12
          Patch Available Patch Available Resolved Resolved
          2h 19m 1 Bikas Saha 19/Apr/13 20:32
          Resolved Resolved Closed Closed
          130d 2h 33m 1 Arun C Murthy 27/Aug/13 23:06

            People

            • Assignee:
              Chris Nauroth
              Reporter:
              Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development