[HADOOP-9488] FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process - ASF JIRA

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 3.0.0
Fix Version/s: 2.1.0-beta
Component/s: fs
Labels:
None

Target Version/s:

3.0.0

Description

FileUtil#createJarWithClassPath always uses System#getEnv for substitution of environment variables in the classpath bundled into the jar manifest. YARN launches container processes with a different set of environment variables, so the method needs to support providing environment variables different from the current process.

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending

Attachments

HADOOP-9488.consolidated.2.patch

19/Apr/13 20:12

17 kB

Chris Nauroth
HADOOP-9488.consolidated.1.patch

19/Apr/13 18:11

17 kB

Chris Nauroth
HADOOP-9488.1.patch

19/Apr/13 18:06

5 kB

Chris Nauroth

Issue Links

is part of

HADOOP-8562 Enhancements to support Hadoop on Windows Server and Windows Azure environments

Closed

relates to

MAPREDUCE-4987 TestMRJobs#testDistributedCache fails on Windows due to classpath problems and unexpected behavior of symlinks

Closed

YARN-593 container launch on Windows does not correctly populate classpath with new process's environment variables and localized resources

Closed

Activity

Descending order - Click to sort in ascending order

Hide

Permalink

Suresh Srinivas added a comment - 24/May/13 23:50

I merged the patch to branch-2.

Show

Suresh Srinivas added a comment - 24/May/13 23:50 I merged the patch to branch-2.

Hide

Permalink

Hudson added a comment - 20/Apr/13 15:07

Integrated in Hadoop-Mapreduce-trunk #1405 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1405/)
~~HADOOP-9488~~. FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996)

Result = SUCCESS
bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996
Files :

/hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

Show

Hudson added a comment - 20/Apr/13 15:07 Integrated in Hadoop-Mapreduce-trunk #1405 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1405/ ) HADOOP-9488 . FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

Hide

Permalink

Hudson added a comment - 20/Apr/13 14:06

Integrated in Hadoop-Hdfs-trunk #1378 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1378/)
~~HADOOP-9488~~. FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996)

Result = FAILURE
bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996
Files :

/hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

Show

Hudson added a comment - 20/Apr/13 14:06 Integrated in Hadoop-Hdfs-trunk #1378 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1378/ ) HADOOP-9488 . FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996) Result = FAILURE bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

Hide

Permalink

Hudson added a comment - 20/Apr/13 11:49

Integrated in Hadoop-Yarn-trunk #189 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/189/)
~~HADOOP-9488~~. FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996)

Result = SUCCESS
bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996
Files :

/hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

Show

Hudson added a comment - 20/Apr/13 11:49 Integrated in Hadoop-Yarn-trunk #189 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/189/ ) HADOOP-9488 . FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

Hide

Permalink

Hadoop QA added a comment - 19/Apr/13 21:37

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12579603/HADOOP-9488.consolidated.2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2461//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2461//console

This message is automatically generated.

Show

Hadoop QA added a comment - 19/Apr/13 21:37 +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579603/HADOOP-9488.consolidated.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2461//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2461//console This message is automatically generated.

Hide

Permalink

Hudson added a comment - 19/Apr/13 20:36

Integrated in Hadoop-trunk-Commit #3636 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3636/)
~~HADOOP-9488~~. FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996)

Result = SUCCESS
bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996
Files :

/hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

Show

Hudson added a comment - 19/Apr/13 20:36 Integrated in Hadoop-trunk-Commit #3636 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3636/ ) HADOOP-9488 . FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

Hide

Permalink

Bikas Saha added a comment - 19/Apr/13 20:32

+1. Committed to trunk.

Show

Bikas Saha added a comment - 19/Apr/13 20:32 +1. Committed to trunk.

Hide

Permalink

Chris Nauroth added a comment - 19/Apr/13 20:12

Updating consolidated patch to check isDirectory just once in the outer loop. (See comments on ~~YARN-593~~.)

Show

Chris Nauroth added a comment - 19/Apr/13 20:12 Updating consolidated patch to check isDirectory just once in the outer loop. (See comments on YARN-593 .)

Hide

Permalink

Hadoop QA added a comment - 19/Apr/13 19:48

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12579576/HADOOP-9488.consolidated.1.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2460//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2460//console

This message is automatically generated.

Show

Hadoop QA added a comment - 19/Apr/13 19:48 +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579576/HADOOP-9488.consolidated.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2460//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2460//console This message is automatically generated.

Hide

Permalink

Chris Nauroth added a comment - 19/Apr/13 18:11

Attaching consolidated patch of all changes previously attached to ~~MAPREDUCE-4987~~, so that we can get a Jenkins run here.

Show

Chris Nauroth added a comment - 19/Apr/13 18:11 Attaching consolidated patch of all changes previously attached to MAPREDUCE-4987 , so that we can get a Jenkins run here.

Hide

Permalink

Chris Nauroth added a comment - 19/Apr/13 18:06

Attaching patch containing just the Hadoop Common portion of the changes. This code has already been reviewed as part of ~~MAPREDUCE-4987~~.

Show

Chris Nauroth added a comment - 19/Apr/13 18:06 Attaching patch containing just the Hadoop Common portion of the changes. This code has already been reviewed as part of MAPREDUCE-4987 .

Hide

Permalink

Chris Nauroth added a comment - 19/Apr/13 17:59

This jira tracks a change that originated in ~~MAPREDUCE-4987~~. The Hadoop Common portion of that patch will be attached here.

Show

Chris Nauroth added a comment - 19/Apr/13 17:59 This jira tracks a change that originated in MAPREDUCE-4987 . The Hadoop Common portion of that patch will be attached here.

People

Assignee:

Chris Nauroth

Reporter:

Chris Nauroth

Votes:

0 Vote for this issue

Watchers:

8 Start watching this issue

Dates

Created:

19/Apr/13 17:57

Updated:

27/Aug/13 23:06

Resolved:

19/Apr/13 20:32

Development

Agile

View on Board