Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5813

YarnChild does not load job.xml with mapreduce.job.classloader=true

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.4.0
    • Component/s: mrv2, task
    • Labels:
      None
    • Target Version/s:

      Description

      YarnChild.main uses JobConf.addResource(String) to load job.xml that relies on class loading. When mapreduce.job.classloader=true the job-speicific part of the class path is separated from CLASSPATH into APP_CLASSPATH. Therefore job.xml is inaccessible for the default class loader. Later writeLocalJobFile overwrites the correct localized job.xml on disk as well.

      This problem is easily avoided by using JobConf.addResource(Path) to read the localized job.xml without relying on class loading.

      1. MAPREDUCE-5813.v01.patch
        1.0 kB
        Gera Shegalov
      2. MAPREDUCE-5813.v02.patch
        5 kB
        Gera Shegalov

        Issue Links

          Activity

          Devaraj K made changes -
          Link This issue is related to MAPREDUCE-5957 [ MAPREDUCE-5957 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1716 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1716/)
          MAPREDUCE-5813. Fix YarnChild to explicitly load job.xml from the local-filesystem, rather than rely on the classpath. Contributed by Gera Shegalov. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1583050)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1716 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1716/ ) MAPREDUCE-5813 . Fix YarnChild to explicitly load job.xml from the local-filesystem, rather than rely on the classpath. Contributed by Gera Shegalov. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1583050 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1742 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1742/)
          MAPREDUCE-5813. Fix YarnChild to explicitly load job.xml from the local-filesystem, rather than rely on the classpath. Contributed by Gera Shegalov. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1583050)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1742 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1742/ ) MAPREDUCE-5813 . Fix YarnChild to explicitly load job.xml from the local-filesystem, rather than rely on the classpath. Contributed by Gera Shegalov. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1583050 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #524 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/524/)
          MAPREDUCE-5813. Fix YarnChild to explicitly load job.xml from the local-filesystem, rather than rely on the classpath. Contributed by Gera Shegalov. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1583050)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #524 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/524/ ) MAPREDUCE-5813 . Fix YarnChild to explicitly load job.xml from the local-filesystem, rather than rely on the classpath. Contributed by Gera Shegalov. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1583050 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5432 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5432/)
          MAPREDUCE-5813. Fix YarnChild to explicitly load job.xml from the local-filesystem, rather than rely on the classpath. Contributed by Gera Shegalov. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1583050)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5432 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5432/ ) MAPREDUCE-5813 . Fix YarnChild to explicitly load job.xml from the local-filesystem, rather than rely on the classpath. Contributed by Gera Shegalov. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1583050 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 2.4.0 [ 12326141 ]
          Resolution Fixed [ 1 ]
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks Gera Shegalov!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks Gera Shegalov !
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12637623/MAPREDUCE-5813.v02.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4468//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4468//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637623/MAPREDUCE-5813.v02.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4468//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4468//console This message is automatically generated.
          Gera Shegalov made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Gera Shegalov made changes -
          Attachment MAPREDUCE-5813.v02.patch [ 12637623 ]
          Hide
          Gera Shegalov added a comment -

          Thanks for catching this, Sangjin Lee. This makes sure that Limits and UGI are also properly configured.

          Adding v02.patch with a unit test.

          Show
          Gera Shegalov added a comment - Thanks for catching this, Sangjin Lee . This makes sure that Limits and UGI are also properly configured. Adding v02.patch with a unit test.
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Arun C Murthy added a comment -

          Agree w/ Sangjin Lee.

          Gera Shegalov - mind updating the patch? Tx.

          Show
          Arun C Murthy added a comment - Agree w/ Sangjin Lee . Gera Shegalov - mind updating the patch? Tx.
          Hide
          Sangjin Lee added a comment -

          The patch looks good to me. The only nit is to add the right job conf before Limits.init() so that Limits.init() can use the right job conf.

          It's nearly the same fix, but how about

          final JobConf job = new JobConf(MRJobConfig.JOB_CONF_FILE);
          Limits.init(job);
          ...
          
          Show
          Sangjin Lee added a comment - The patch looks good to me. The only nit is to add the right job conf before Limits.init() so that Limits.init() can use the right job conf. It's nearly the same fix, but how about final JobConf job = new JobConf(MRJobConfig.JOB_CONF_FILE); Limits.init(job); ...
          Hide
          Gera Shegalov added a comment -

          The TestMRAppMaster NPE are related to MAPREDUCE-5805. This JIRA touches only YarnChild for Map/ReduceTask

          Show
          Gera Shegalov added a comment - The TestMRAppMaster NPE are related to MAPREDUCE-5805 . This JIRA touches only YarnChild for Map/ReduceTask
          Gera Shegalov made changes -
          Link This issue is broken by MAPREDUCE-5805 [ MAPREDUCE-5805 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12637128/MAPREDUCE-5813.v01.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:

          org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4464//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4464//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637128/MAPREDUCE-5813.v01.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4464//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4464//console This message is automatically generated.
          Gera Shegalov made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Gera Shegalov made changes -
          Attachment MAPREDUCE-5813.v01.patch [ 12637128 ]
          Hide
          Gera Shegalov added a comment -

          v01 patch for review.

          Show
          Gera Shegalov added a comment - v01 patch for review.
          Gera Shegalov made changes -
          Field Original Value New Value
          Assignee Gera Shegalov [ jira.shegalov ]
          Gera Shegalov created issue -

            People

            • Assignee:
              Gera Shegalov
              Reporter:
              Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development