Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5459

Update the doc of running MRv1 examples jar on YARN

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.2.0
    • Component/s: None
    • Labels:
      None

      Description

      In addition to adding two env vars: HADOOP_USER_CLASSPATH_FIRST and HADOOP_CLASSPATH, we still need to add

          <property>
              <name>mapreduce.job.user.classpath.first</name>
              <value>true</value>
          </property>
      

      in mapred-site.xml to make sure that the MRv1 examples jar runs correctly on YARN. Some examples will use Java reflection to find the classes in the examples jar dynamically when they are running. With this configuration, the MRv1 examples jar will appear before the MRv2 examples jar in CLASSPATH of the processes in YARN containers. Therefore, the classes found via reflection will be picked from MRv1 examples jar instead of MRv2 examples jar as well.

      MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm needs to be updated to document this.

        Issue Links

          Activity

          Hide
          zjshen Zhijie Shen added a comment -

          Created a patch to update MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm

          Show
          zjshen Zhijie Shen added a comment - Created a patch to update MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12598061/MAPREDUCE-5459.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3942//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3942//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12598061/MAPREDUCE-5459.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3942//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3942//console This message is automatically generated.
          Hide
          acmurthy Arun C Murthy added a comment -

          I just committed this. Thanks Zhijie Shen!

          Show
          acmurthy Arun C Murthy added a comment - I just committed this. Thanks Zhijie Shen !
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #4519 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4519/)
          MAPREDUCE-5459. Update documentation on how to run MRv1 examples on YARN. Contributed by Zhijie Shen. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528626)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4519 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4519/ ) MAPREDUCE-5459 . Update documentation on how to run MRv1 examples on YARN. Contributed by Zhijie Shen. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528626 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #351 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/351/)
          MAPREDUCE-5459. Update documentation on how to run MRv1 examples on YARN. Contributed by Zhijie Shen. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528626)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #351 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/351/ ) MAPREDUCE-5459 . Update documentation on how to run MRv1 examples on YARN. Contributed by Zhijie Shen. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528626 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1541 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1541/)
          MAPREDUCE-5459. Update documentation on how to run MRv1 examples on YARN. Contributed by Zhijie Shen. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528626)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1541 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1541/ ) MAPREDUCE-5459 . Update documentation on how to run MRv1 examples on YARN. Contributed by Zhijie Shen. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528626 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1567 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1567/)
          MAPREDUCE-5459. Update documentation on how to run MRv1 examples on YARN. Contributed by Zhijie Shen. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528626)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1567 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1567/ ) MAPREDUCE-5459 . Update documentation on how to run MRv1 examples on YARN. Contributed by Zhijie Shen. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528626 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing old tickets that are already part of a release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing old tickets that are already part of a release.

            People

            • Assignee:
              zjshen Zhijie Shen
              Reporter:
              zjshen Zhijie Shen
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development