Hadoop Common
  1. Hadoop Common
  2. HADOOP-9488

FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process

        Details

        • Type: Bug Bug
        • Status: Closed
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: 3.0.0-alpha1
        • Fix Version/s: 2.1.0-beta
        • Component/s: fs
        • Labels:
          None
        • Target Version/s:

          Description

          FileUtil#createJarWithClassPath always uses System#getEnv for substitution of environment variables in the classpath bundled into the jar manifest. YARN launches container processes with a different set of environment variables, so the method needs to support providing environment variables different from the current process.

          1. Text File
            HADOOP-9488.1.patch
            5 kB
            Chris Nauroth
          2. Text File
            HADOOP-9488.consolidated.1.patch
            17 kB
            Chris Nauroth
          3. Text File
            HADOOP-9488.consolidated.2.patch
            17 kB
            Chris Nauroth

            Issue Links

            is part of

            New Feature - A new feature of the product, which has yet to be developed. HADOOP-8562 Enhancements to support Hadoop on Windows Server and Windows Azure environments

            • Major - Major loss of function.
            • Closed
            relates to

            Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-4987 TestMRJobs#testDistributedCache fails on Windows due to classpath problems and unexpected behavior of symlinks

            • Major - Major loss of function.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. YARN-593 container launch on Windows does not correctly populate classpath with new process's environment variables and localized resources

            • Major - Major loss of function.
            • Closed

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              Chris Nauroth added a comment -

              This jira tracks a change that originated in MAPREDUCE-4987. The Hadoop Common portion of that patch will be attached here.

              Show
              Chris Nauroth added a comment - This jira tracks a change that originated in MAPREDUCE-4987 . The Hadoop Common portion of that patch will be attached here.
              Hide
              Permalink
              Chris Nauroth added a comment -

              Attaching patch containing just the Hadoop Common portion of the changes. This code has already been reviewed as part of MAPREDUCE-4987.

              Show
              Chris Nauroth added a comment - Attaching patch containing just the Hadoop Common portion of the changes. This code has already been reviewed as part of MAPREDUCE-4987 .
              Hide
              Permalink
              Chris Nauroth added a comment -

              Attaching consolidated patch of all changes previously attached to MAPREDUCE-4987, so that we can get a Jenkins run here.

              Show
              Chris Nauroth added a comment - Attaching consolidated patch of all changes previously attached to MAPREDUCE-4987 , so that we can get a Jenkins run here.
              Hide
              Permalink
              Hadoop QA added a comment -

              +1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12579576/HADOOP-9488.consolidated.1.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 2 new or modified test files.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2460//testReport/
              Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2460//console

              This message is automatically generated.

              Show
              Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579576/HADOOP-9488.consolidated.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2460//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2460//console This message is automatically generated.
              Hide
              Permalink
              Chris Nauroth added a comment -

              Updating consolidated patch to check isDirectory just once in the outer loop. (See comments on YARN-593.)

              Show
              Chris Nauroth added a comment - Updating consolidated patch to check isDirectory just once in the outer loop. (See comments on YARN-593 .)
              Hide
              Permalink
              Bikas Saha added a comment -

              +1. Committed to trunk.

              Show
              Bikas Saha added a comment - +1. Committed to trunk.
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-trunk-Commit #3636 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3636/)
              HADOOP-9488. FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996)

              Result = SUCCESS
              bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
              Show
              Hudson added a comment - Integrated in Hadoop-trunk-Commit #3636 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3636/ ) HADOOP-9488 . FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
              Hide
              Permalink
              Hadoop QA added a comment -

              +1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12579603/HADOOP-9488.consolidated.2.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 2 new or modified test files.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2461//testReport/
              Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2461//console

              This message is automatically generated.

              Show
              Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579603/HADOOP-9488.consolidated.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2461//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2461//console This message is automatically generated.
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Yarn-trunk #189 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/189/)
              HADOOP-9488. FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996)

              Result = SUCCESS
              bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
              Show
              Hudson added a comment - Integrated in Hadoop-Yarn-trunk #189 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/189/ ) HADOOP-9488 . FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Hdfs-trunk #1378 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1378/)
              HADOOP-9488. FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996)

              Result = FAILURE
              bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
              Show
              Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1378 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1378/ ) HADOOP-9488 . FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996) Result = FAILURE bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Mapreduce-trunk #1405 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1405/)
              HADOOP-9488. FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996)

              Result = SUCCESS
              bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
              Show
              Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1405 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1405/ ) HADOOP-9488 . FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process (Chris Nauroth via bikas) (Revision 1469996) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469996 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
              Hide
              Permalink
              Suresh Srinivas added a comment -

              I merged the patch to branch-2.

              Show
              Suresh Srinivas added a comment - I merged the patch to branch-2.

                People

                • Assignee:
                  Chris Nauroth
                  Reporter:
                  Chris Nauroth
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  8 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development