Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6704

Update the documents to run MapReduce application

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: documentation
    • Labels:
      None

      Description

      Container fail to launch for mapred application.
      As part for launch script HADOOP_MAPRED_HOME default value is not set .After https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c HADOOP_MAPRED_HOME is not able to get from builder.environment() since DefaultContainerExecutor#buildCommandExecutor sets inherit to false.

      16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with state FAILED due to: Application application_1462155939310_0004 failed 2 times due to AM Container for appattempt_1462155939310_0004_000002 exited with  exitCode: 1
      Failing this attempt.Diagnostics: Exception from container-launch.
      Container id: container_1462155939310_0004_02_000001
      Exit code: 1
      Stack trace: ExitCodeException exitCode=1:
              at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
              at org.apache.hadoop.util.Shell.run(Shell.java:850)
              at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
              at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      
      Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
      Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier; support was removed in 8.0
      Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
      
      Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
      Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier; support was removed in 8.0
      
      1. 0001-MAPREDUCE-6704.patch
        4 kB
        Bibin A Chundatt
      2. 0001-YARN-5026.patch
        4 kB
        Bibin A Chundatt
      3. ClusterSetup.html
        54 kB
        Bibin A Chundatt
      4. container-whitelist-env-wip.patch
        13 kB
        Bibin A Chundatt
      5. MAPREDUCE-6704.0002.patch
        3 kB
        Bibin A Chundatt
      6. MAPREDUCE-6704.0002.patch
        3 kB
        Bibin A Chundatt
      7. MR-6704-branch2.8.tar.gz
        14 kB
        Zhankun Tang
      8. MR-6704-trunk.tar.gz
        5 kB
        Zhankun Tang
      9. MR-6704-trunk-tempPatch.tar.gz
        20 kB
        Zhankun Tang
      10. SingleCluster.html
        31 kB
        Bibin A Chundatt
      11. temp.patch
        2 kB
        Bibin A Chundatt

        Issue Links

          Activity

          Hide
          Naganarasimha Naganarasimha G R added a comment -
          Show
          Naganarasimha Naganarasimha G R added a comment - cc Kousuke Saruta & Akira Ajisaka
          Hide
          sarutak Kousuke Saruta added a comment -

          Thanks for the report Bibin A Chundatt.
          I've investigated a little this issue.
          I guess the change in https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c causes this issue rather than YARN-4630 if the reason why the MR job fails is as you mention.

          Show
          sarutak Kousuke Saruta added a comment - Thanks for the report Bibin A Chundatt . I've investigated a little this issue. I guess the change in https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c causes this issue rather than YARN-4630 if the reason why the MR job fails is as you mention.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Kousuke Saruta
          Thank you for correcting me. Will update description too.
          cc Robert Kanter

          Show
          bibinchundatt Bibin A Chundatt added a comment - Kousuke Saruta Thank you for correcting me. Will update description too. cc Robert Kanter
          Hide
          rkanter Robert Kanter added a comment -

          HADOOP_MAPRED_HOME shouldn't be necessary if you're running MR from a tarball in HDFS. If you need to add HADOOP_MAPRED_HOME, then I believe you should be able to add it to the classpath for MR jobs via mapreduce.admin.user.env.

          Show
          rkanter Robert Kanter added a comment - HADOOP_MAPRED_HOME shouldn't be necessary if you're running MR from a tarball in HDFS. If you need to add HADOOP_MAPRED_HOME , then I believe you should be able to add it to the classpath for MR jobs via mapreduce.admin.user.env .
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Currently we have changed the default behaviour. In launch_container.sh shouldn't we be adding the default value too rt??

          export HADOOP_MAPRED_HOME={HADOOP_MAPRED_HOME:-xxxxx}
          

          So that without setting mapreduce.admin.user.env also we would get the old behaviour.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Currently we have changed the default behaviour. In launch_container.sh shouldn't we be adding the default value too rt?? export HADOOP_MAPRED_HOME={HADOOP_MAPRED_HOME:-xxxxx} So that without setting mapreduce.admin.user.env also we would get the old behaviour.
          Hide
          rkanter Robert Kanter added a comment -

          Containers shouldn't be relying on getting their environment variables from the NodeManager. For compatibility, the change in https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c was only done in trunk. The equivalent change in branch-2, https://github.com/apache/hadoop/commit/ac8fb579c6058fec60caf30682f902413d68edf3, does not remove all of the environment variables.

          Show
          rkanter Robert Kanter added a comment - Containers shouldn't be relying on getting their environment variables from the NodeManager. For compatibility, the change in https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c was only done in trunk. The equivalent change in branch-2, https://github.com/apache/hadoop/commit/ac8fb579c6058fec60caf30682f902413d68edf3 , does not remove all of the environment variables.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Robert Kanter
          Thank you for clarification
          As mentioned above after adding mapreduce.admin.user.env and yarn.app.mapreduce.am.env it works fine.
          IMHO it will be good to make it work without setting additional parameters too. Attaching patch for the same.

          Could you please point me to the JIRA for which change was done to understand further.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Robert Kanter Thank you for clarification As mentioned above after adding mapreduce.admin.user.env and yarn.app.mapreduce.am.env it works fine. IMHO it will be good to make it work without setting additional parameters too. Attaching patch for the same. Could you please point me to the JIRA for which change was done to understand further.
          Hide
          sunilg Sunil G added a comment -

          Bibin A Chundatt / Robert Kanter

          Is it possible that we can set default value to mapreduce.admin.user.env and yarn.app.mapreduce.am.env. By default HADOOP_MAPRED_HOME can be set as HADOOP_HOME. I guess this can be helpful, and not feeling this as security issue. Robert Kanter, do you any security issue if we set a default value?

          Show
          sunilg Sunil G added a comment - Bibin A Chundatt / Robert Kanter Is it possible that we can set default value to mapreduce.admin.user.env and yarn.app.mapreduce.am.env . By default HADOOP_MAPRED_HOME can be set as HADOOP_HOME. I guess this can be helpful, and not feeling this as security issue. Robert Kanter , do you any security issue if we set a default value?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sunil G
          Yes,Was considering this as second solution. As per the offline discussion with Rohith Sharma K S Yarn shouldn't have any MR-specific configs so setting the default value as HADOOP_COMMON_HOME for HADOOP_MAPRED_HOME should fix the problem.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Sunil G Yes,Was considering this as second solution. As per the offline discussion with Rohith Sharma K S Yarn shouldn't have any MR-specific configs so setting the default value as HADOOP_COMMON_HOME for HADOOP_MAPRED_HOME should fix the problem.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Bibin A Chundatt, would you like to update the patch to implement the approach (setting HADOOP_MAPRED_HOME to HADOOP_COMMON_HOME)?

          One thing to note is that if the default is defined on the admin env and the user overrides it on the non-admin env things break because YARN would simply concatenate the values (see YARN-4789 and MAPREDUCE-6491). So until that is fixed it might be better to use mapred.child.env over mapreduce.admin.user.env.

          Show
          sjlee0 Sangjin Lee added a comment - Bibin A Chundatt , would you like to update the patch to implement the approach (setting HADOOP_MAPRED_HOME to HADOOP_COMMON_HOME)? One thing to note is that if the default is defined on the admin env and the user overrides it on the non-admin env things break because YARN would simply concatenate the values (see YARN-4789 and MAPREDUCE-6491 ). So until that is fixed it might be better to use mapred.child.env over mapreduce.admin.user.env .
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          we should move this to Mapred project, if fix is on map reduce related configuration/env

          Show
          Naganarasimha Naganarasimha G R added a comment - we should move this to Mapred project, if fix is on map reduce related configuration/env
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sangjin Lee/Naganarasimha G R
          Will update the patch soon after moving to mapreduce

          Show
          bibinchundatt Bibin A Chundatt added a comment - Sangjin Lee / Naganarasimha G R Will update the patch soon after moving to mapreduce
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 33s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 6m 41s trunk passed
          +1 compile 1m 34s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 1m 26s trunk passed
          +1 javadoc 0m 40s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 0m 43s the patch passed
          +1 compile 1m 35s the patch passed
          +1 javac 1m 35s the patch passed
          -1 checkstyle 0m 31s hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 2 new + 621 unchanged - 2 fixed = 623 total (was 623)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 1m 33s the patch passed
          +1 javadoc 0m 32s the patch passed
          +1 unit 2m 1s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 8m 48s hadoop-mapreduce-client-app in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          31m 0s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809399/0001-MAPREDUCE-6704.patch
          JIRA Issue MAPREDUCE-6704
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2887d3993199 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9581fb7
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6548/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6548/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6548/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6548/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 33s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 6m 41s trunk passed +1 compile 1m 34s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 1m 26s trunk passed +1 javadoc 0m 40s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 0m 43s the patch passed +1 compile 1m 35s the patch passed +1 javac 1m 35s the patch passed -1 checkstyle 0m 31s hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 2 new + 621 unchanged - 2 fixed = 623 total (was 623) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 21s the patch passed -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 33s the patch passed +1 javadoc 0m 32s the patch passed +1 unit 2m 1s hadoop-mapreduce-client-core in the patch passed. +1 unit 8m 48s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 31m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809399/0001-MAPREDUCE-6704.patch JIRA Issue MAPREDUCE-6704 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2887d3993199 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9581fb7 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6548/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6548/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6548/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6548/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Bibin A Chundatt thanks for your contribution.

          + "HADOOP_MAPRED_HOME=" + Apps.crossPlatformify("HADOOP_COMMON_HOME");

          Why not making {{ "HADOOP_MAPRED_HOME="Apps.crossPlatformify("HADOOP_MAPRED_HOME");}}? I think it's more straight forward and better to understand its behaviour.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Bibin A Chundatt thanks for your contribution. + "HADOOP_MAPRED_HOME=" + Apps.crossPlatformify("HADOOP_COMMON_HOME"); Why not making {{ "HADOOP_MAPRED_HOME="Apps.crossPlatformify("HADOOP_MAPRED_HOME");}}? I think it's more straight forward and better to understand its behaviour.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Also, after changing default value of properties, could you update ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml?

          Show
          ozawa Tsuyoshi Ozawa added a comment - Also, after changing default value of properties, could you update ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          By default HADOOP_MAPRED_HOME can be set as HADOOP_HOME. I guess this can be helpful, and not feeling this as security issue.

          Agreed. If we are going to do this, we need to revert the documentation of MAPREDUCE-6702 and update the release note. Would you revert the documentation change in this issue?

          Show
          ajisakaa Akira Ajisaka added a comment - By default HADOOP_MAPRED_HOME can be set as HADOOP_HOME. I guess this can be helpful, and not feeling this as security issue. Agreed. If we are going to do this, we need to revert the documentation of MAPREDUCE-6702 and update the release note. Would you revert the documentation change in this issue?
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Akira Ajisaka

          Agreed. If we are going to do this, we need to revert the documentation of MAPREDUCE-6702 and update the release note. Would you revert the documentation change in this issue?

          My previous comment means I don't agree with the point. My thought is that it is not straight forward, and confusing. Hence, I think it should be {{ "HADOOP_MAPRED_HOME="Apps.crossPlatformify("HADOOP_MAPRED_HOME");}}. Could you tell me your thoughts?

          Show
          ozawa Tsuyoshi Ozawa added a comment - Akira Ajisaka Agreed. If we are going to do this, we need to revert the documentation of MAPREDUCE-6702 and update the release note. Would you revert the documentation change in this issue? My previous comment means I don't agree with the point. My thought is that it is not straight forward, and confusing. Hence, I think it should be {{ "HADOOP_MAPRED_HOME="Apps.crossPlatformify("HADOOP_MAPRED_HOME");}}. Could you tell me your thoughts?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          IIUC, "HADOOP_MAPRED_HOME=" + Apps.crossPlatformify("HADOOP_MAPRED_HOME") does not fix the problem because HADOOP_MAPRED_HOME is not in yarn.nodemanager.env-whitelist by default (i.e. Container cannot inherit the NodeManager's environment variable HADOOP_MAPRED_HOME by default).

          Show
          ajisakaa Akira Ajisaka added a comment - IIUC, "HADOOP_MAPRED_HOME=" + Apps.crossPlatformify("HADOOP_MAPRED_HOME") does not fix the problem because HADOOP_MAPRED_HOME is not in yarn.nodemanager.env-whitelist by default (i.e. Container cannot inherit the NodeManager's environment variable HADOOP_MAPRED_HOME by default).
          Hide
          aw Allen Wittenauer added a comment -

          There has been no guarantee that HADOOP_MAPRED_HOME is the same as HADOOP_COMMON_HOME since 0.22. See libexec/hadoop-layout.sh.example.

          Show
          aw Allen Wittenauer added a comment - There has been no guarantee that HADOOP_MAPRED_HOME is the same as HADOOP_COMMON_HOME since 0.22. See libexec/hadoop-layout.sh.example.
          Hide
          bibinchundatt Bibin A Chundatt added a comment - - edited

          Allen Wittenauer/Akira Ajisaka
          Sorry for the delay.

          As already mentioned the HADOOP_MAPRED_HOME is not part of env since not inherited to launcher.Adding

          yarn.nodemanager.env-whitelist="JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME"
          

          in yarn-site.xml of NM does solve the problem. But since its a mapreduce property its not correct to add to the whitelist of yarn .

          IIUU HADOOP_MAPRED_HOME for default installation will be same as HADOOP_COMMON_HOME .
          For layout change we can configure explicity mapred.child.env paths.

          Till jira is fixed ,to make mapreduce job run add params as below

          -Dmapred.child.env="HADOOP_MAPRED_HOME={{HADOOP_COMMON_HOME}}" -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME={{HADOOP_COMMON_HOME}}"
          

          Open to any other solution.

          Show
          bibinchundatt Bibin A Chundatt added a comment - - edited Allen Wittenauer / Akira Ajisaka Sorry for the delay. As already mentioned the HADOOP_MAPRED_HOME is not part of env since not inherited to launcher.Adding yarn.nodemanager.env-whitelist= "JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME" in yarn-site.xml of NM does solve the problem. But since its a mapreduce property its not correct to add to the whitelist of yarn . IIUU HADOOP_MAPRED_HOME for default installation will be same as HADOOP_COMMON_HOME . For layout change we can configure explicity mapred.child.env paths. Till jira is fixed ,to make mapreduce job run add params as below -Dmapred.child.env= "HADOOP_MAPRED_HOME={{HADOOP_COMMON_HOME}}" -Dyarn.app.mapreduce.am.env= "HADOOP_MAPRED_HOME={{HADOOP_COMMON_HOME}}" Open to any other solution.
          Hide
          aw Allen Wittenauer added a comment -

          But since its a mapreduce property its not correct to add to the whitelist of yarn .

          Why can't MR add it itself?

          But since its a mapreduce property its not correct to add to the whitelist of yarn .

          Who cares? It's all Apache Hadoop. Users have an expectation that this stuff will work out of the box and be consistent. They aren't going to say "Oh, of course this is completely broken! That part over there is YARN and that part over there is MAPREDUCE!"

          IIUU HADOOP_MAPRED_HOME for default installation will be same as HADOOP_COMMON_HOME .

          It's irrelevant. HADOOP_MAPRED_HOME is where the mapred stuff is guaranteed to be. There is no guarantee that HADOOP_COMMON_HOME is going to work. That's why there is a separate variable for it instead of just using HADOOP_COMMON_HOME everywhere. Either this should be HADOOP_MAPRED_HOME or set nothing at all.

          Show
          aw Allen Wittenauer added a comment - But since its a mapreduce property its not correct to add to the whitelist of yarn . Why can't MR add it itself? But since its a mapreduce property its not correct to add to the whitelist of yarn . Who cares? It's all Apache Hadoop. Users have an expectation that this stuff will work out of the box and be consistent . They aren't going to say "Oh, of course this is completely broken! That part over there is YARN and that part over there is MAPREDUCE!" IIUU HADOOP_MAPRED_HOME for default installation will be same as HADOOP_COMMON_HOME . It's irrelevant. HADOOP_MAPRED_HOME is where the mapred stuff is guaranteed to be. There is no guarantee that HADOOP_COMMON_HOME is going to work. That's why there is a separate variable for it instead of just using HADOOP_COMMON_HOME everywhere. Either this should be HADOOP_MAPRED_HOME or set nothing at all.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          0001-YARN-5026.patch implementation is based on HADOOP_MAPRED_HOME added to whitelist.

          Show
          bibinchundatt Bibin A Chundatt added a comment - 0001- YARN-5026 .patch implementation is based on HADOOP_MAPRED_HOME added to whitelist.
          Hide
          tangzhankun Zhankun Tang added a comment - - edited

          I encountered this issue too. Quite shocked that commit "9d4d30243b0fc" get merged without basic MRv2 test and even no YARN JIRA for it!
          And it's strange that I still fail to run MRv2 application as well as Dockerized MRv2 application after tried the temporary solution mentioned by Bibin A Chundatt. YARN patch also not works for me.

          Show
          tangzhankun Zhankun Tang added a comment - - edited I encountered this issue too. Quite shocked that commit "9d4d30243b0fc" get merged without basic MRv2 test and even no YARN JIRA for it! And it's strange that I still fail to run MRv2 application as well as Dockerized MRv2 application after tried the temporary solution mentioned by Bibin A Chundatt. YARN patch also not works for me.
          Hide
          bibinchundatt Bibin A Chundatt added a comment - - edited

          Zhankun Tang

          In nodemanager we can add the below config. Last time i checked below solution was working fine.

          yarn.nodemanager.env-whitelist=
          "JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME"

          Show
          bibinchundatt Bibin A Chundatt added a comment - - edited Zhankun Tang In nodemanager we can add the below config. Last time i checked below solution was working fine. yarn.nodemanager.env-whitelist= "JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME"
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 6m 58s trunk passed
          +1 compile 1m 33s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 1m 24s trunk passed
          +1 javadoc 0m 36s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 43s the patch passed
          +1 compile 1m 29s the patch passed
          +1 javac 1m 29s the patch passed
          -1 checkstyle 0m 31s hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 2 new + 622 unchanged - 2 fixed = 624 total (was 624)
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 36s the patch passed
          +1 javadoc 0m 31s the patch passed
          +1 unit 2m 2s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 8m 54s hadoop-mapreduce-client-app in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          30m 58s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809399/0001-MAPREDUCE-6704.patch
          JIRA Issue MAPREDUCE-6704
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a9335e2f557f 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 964e546
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6730/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6730/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6730/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 6m 58s trunk passed +1 compile 1m 33s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 1m 24s trunk passed +1 javadoc 0m 36s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 43s the patch passed +1 compile 1m 29s the patch passed +1 javac 1m 29s the patch passed -1 checkstyle 0m 31s hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 2 new + 622 unchanged - 2 fixed = 624 total (was 624) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 36s the patch passed +1 javadoc 0m 31s the patch passed +1 unit 2m 2s hadoop-mapreduce-client-core in the patch passed. +1 unit 8m 54s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 30m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809399/0001-MAPREDUCE-6704.patch JIRA Issue MAPREDUCE-6704 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a9335e2f557f 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 964e546 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6730/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6730/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6730/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          Quite shocked that commit "9d4d30243b0fc" get merged without basic MRv2 test and even no YARN JIRA for it!

          Zhankun Tang, commit "9d4d30243b0fc" when in to fix a security problem. That's why there's no JIRA and why it's shrouded in mystery.

          Who cares? It's all Apache Hadoop. Users have an expectation that this stuff will work out of the box and be consistent. They aren't going to say "Oh, of course this is completely broken! That part over there is YARN and that part over there is MAPREDUCE!"

          Allen Wittenauer, my original patch for "9d4d30243b0fc" did add HADOOP_MAPRED_HOME to Yarn and things did work out of the box. There was pushback to remove it because of the desire to keep Yarn and MR separate. Given the difficulty people seem to be having and that there doesn't seem to be a single fix that works for everyone for some reason, perhaps we should revisit that decision?

          Show
          rkanter Robert Kanter added a comment - Quite shocked that commit "9d4d30243b0fc" get merged without basic MRv2 test and even no YARN JIRA for it! Zhankun Tang , commit "9d4d30243b0fc" when in to fix a security problem. That's why there's no JIRA and why it's shrouded in mystery. Who cares? It's all Apache Hadoop. Users have an expectation that this stuff will work out of the box and be consistent. They aren't going to say "Oh, of course this is completely broken! That part over there is YARN and that part over there is MAPREDUCE!" Allen Wittenauer , my original patch for "9d4d30243b0fc" did add HADOOP_MAPRED_HOME to Yarn and things did work out of the box. There was pushback to remove it because of the desire to keep Yarn and MR separate. Given the difficulty people seem to be having and that there doesn't seem to be a single fix that works for everyone for some reason, perhaps we should revisit that decision?
          Hide
          yzhangal Yongjun Zhang added a comment -

          HI Guys,

          Thanks for working on this issue here. Wonder if we had a consensus about the solution? Can we document the workaround before the solution is in place?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - HI Guys, Thanks for working on this issue here. Wonder if we had a consensus about the solution? Can we document the workaround before the solution is in place? Thanks.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Per the discussion above, I added config

          <property>
          <name>yarn.nodemanager.env-whitelist</name>
          <value>=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
          </property>
          

          to yarn-site.xml

          and ran into the following issue

          Container exited with a non-zero exit code 127. Last 4096 bytes of stderr :
          /bin/bash: /bin/java: No such file or directory
          

          Then I created a soft link from the java I am using to /bin/java, and it works now.

          It seems the code has a hardcoded dependency on the existence of /bin/java. Can we find the java in the path setting instead?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Per the discussion above, I added config <property> <name>yarn.nodemanager.env-whitelist</name> <value>=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value> </property> to yarn-site.xml and ran into the following issue Container exited with a non-zero exit code 127. Last 4096 bytes of stderr : /bin/bash: /bin/java: No such file or directory Then I created a soft link from the java I am using to /bin/java, and it works now. It seems the code has a hardcoded dependency on the existence of /bin/java. Can we find the java in the path setting instead? Thanks.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Yongjun Zhang
          Looks like you have made wrong configuration

          <value>=JAVA_HOME
          

          Could please remove = and try again. I thought of documenting the white list parameter configuration and closing the issue waiting for +1 which solution to take.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Yongjun Zhang Looks like you have made wrong configuration <value>=JAVA_HOME Could please remove = and try again. I thought of documenting the white list parameter configuration and closing the issue waiting for +1 which solution to take.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks Bibin A Chundatt, good catch, and sorry for missing that earlier. I tried again after removing "+", it runs.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks Bibin A Chundatt , good catch, and sorry for missing that earlier. I tried again after removing "+", it runs.
          Hide
          andrew.wang Andrew Wang added a comment -

          Folks, is there any progress we can make on this JIRA? That this doesn't work out of the box anymore has been very surprising to our users. I'd like to get it fixed for alpha2 if possible.

          Show
          andrew.wang Andrew Wang added a comment - Folks, is there any progress we can make on this JIRA? That this doesn't work out of the box anymore has been very surprising to our users. I'd like to get it fixed for alpha2 if possible.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Robert Kanter

          Given the difficulty people seem to be having and that there doesn't seem to be a single fix that works for everyone for some reason, perhaps we should revisit that decision?

          IMHO we have to rethink of the same.

          Summarizing discussion and solutions

          1. Add HADOOP_MAPRED_HOME=HADOOP_COMMON_HOME in opts.But its not mandatory that MAPRED_HOME=HADOOP_COMMON_HOME
          2. Add HADOOP_MAPRED_HOME to Yarn.Since we want to keep YARN and MAPRED separate initial solution was not accepted.
          3. Add documentation to configure yarn.nodemanager.env-whitelist in nodemanager to run mapred application
            <property>
            <name>yarn.nodemanager.env-whitelist</name>
            <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
            </property>
            

            Waiting for +1 for any one of above solution or inputs for any other approach

          Show
          bibinchundatt Bibin A Chundatt added a comment - Robert Kanter Given the difficulty people seem to be having and that there doesn't seem to be a single fix that works for everyone for some reason, perhaps we should revisit that decision? IMHO we have to rethink of the same. Summarizing discussion and solutions Add HADOOP_MAPRED_HOME=HADOOP_COMMON_HOME in opts.But its not mandatory that MAPRED_HOME=HADOOP_COMMON_HOME Add HADOOP_MAPRED_HOME to Yarn.Since we want to keep YARN and MAPRED separate initial solution was not accepted. Add documentation to configure yarn.nodemanager.env-whitelist in nodemanager to run mapred application <property> <name>yarn.nodemanager.env-whitelist</name> <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value> </property> Waiting for +1 for any one of above solution or inputs for any other approach
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Solution 4:
          We should add option in YARN to support sending addition whitelist ENV as part of ContainerLaunchContext. Addition patch for solution 4 also container-whitelist-env-wip.patch . created a jira YARN-5771 also.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Solution 4: We should add option in YARN to support sending addition whitelist ENV as part of ContainerLaunchContext. Addition patch for solution 4 also container-whitelist-env-wip.patch . created a jira YARN-5771 also.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 40s Maven dependency ordering for branch
          +1 mvninstall 6m 52s trunk passed
          +1 compile 6m 45s trunk passed
          +1 checkstyle 1m 36s trunk passed
          +1 mvnsite 2m 46s trunk passed
          +1 mvneclipse 1m 21s trunk passed
          +1 findbugs 4m 18s trunk passed
          +1 javadoc 1m 48s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 12s the patch passed
          +1 compile 6m 44s the patch passed
          +1 cc 6m 44s the patch passed
          +1 javac 6m 44s the patch passed
          -1 checkstyle 1m 38s root: The patch generated 8 new + 865 unchanged - 3 fixed = 873 total (was 868)
          +1 mvnsite 2m 45s the patch passed
          +1 mvneclipse 1m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 1s the patch passed
          +1 javadoc 1m 49s the patch passed
          +1 unit 0m 24s hadoop-yarn-api in the patch passed.
          +1 unit 2m 16s hadoop-yarn-common in the patch passed.
          -1 unit 14m 43s hadoop-yarn-server-nodemanager in the patch failed.
          +1 unit 2m 27s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 8m 31s hadoop-mapreduce-client-app in the patch passed.
          +1 unit 111m 23s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          189m 53s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834840/container-whitelist-env-wip.patch
          JIRA Issue MAPREDUCE-6704
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux bace1b096a29 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d4725bf
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6771/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6771/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6771/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6771/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6771/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 40s Maven dependency ordering for branch +1 mvninstall 6m 52s trunk passed +1 compile 6m 45s trunk passed +1 checkstyle 1m 36s trunk passed +1 mvnsite 2m 46s trunk passed +1 mvneclipse 1m 21s trunk passed +1 findbugs 4m 18s trunk passed +1 javadoc 1m 48s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 12s the patch passed +1 compile 6m 44s the patch passed +1 cc 6m 44s the patch passed +1 javac 6m 44s the patch passed -1 checkstyle 1m 38s root: The patch generated 8 new + 865 unchanged - 3 fixed = 873 total (was 868) +1 mvnsite 2m 45s the patch passed +1 mvneclipse 1m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 1s the patch passed +1 javadoc 1m 49s the patch passed +1 unit 0m 24s hadoop-yarn-api in the patch passed. +1 unit 2m 16s hadoop-yarn-common in the patch passed. -1 unit 14m 43s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 2m 27s hadoop-mapreduce-client-core in the patch passed. +1 unit 8m 31s hadoop-mapreduce-client-app in the patch passed. +1 unit 111m 23s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 189m 53s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834840/container-whitelist-env-wip.patch JIRA Issue MAPREDUCE-6704 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux bace1b096a29 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d4725bf Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6771/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6771/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6771/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6771/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6771/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment - - edited

          +1 to #3, but making it the default, not just documenting it. I agree with Allen Wittenauer that a little pollution is OK if it makes the out-of-box experience significantly better. And I don't think spilling over an environment variable is so bad anyway.

          #1 has several issues. In addition to what was already stated, the vars are evaluated on the client, so if the client doesn't have the same paths as the NM hosts, it won't work.

          #2 is a code-level pollution, which is a more serious thing than #3.

          #4 doesn't sound like the right idea to me. Users don't need to control the whitelisting on a per-job basis, so it's solving a problem that doesn't exist. It also has some security implications that need to be carefully considered.

          Show
          templedf Daniel Templeton added a comment - - edited +1 to #3, but making it the default , not just documenting it. I agree with Allen Wittenauer that a little pollution is OK if it makes the out-of-box experience significantly better. And I don't think spilling over an environment variable is so bad anyway. #1 has several issues. In addition to what was already stated, the vars are evaluated on the client, so if the client doesn't have the same paths as the NM hosts, it won't work. #2 is a code-level pollution, which is a more serious thing than #3. #4 doesn't sound like the right idea to me. Users don't need to control the whitelisting on a per-job basis, so it's solving a problem that doesn't exist. It also has some security implications that need to be carefully considered.
          Hide
          templedf Daniel Templeton added a comment -

          As if we're not split enough on this decision, let me throw in one more option.

          The final and correct answer is that users should set up distributed cache deployment (per https://hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html). At that point things just work, and upgrades become simpler. The problem is that distributed cache deployment isn't an option for out of the box.

          Spark does not have the problem we're trying to solve here. Every time you submit a Spark application, they ship the assembly JAR via the distributed cache. If we want to push users to distributed cache deployment, maybe the way to solve the out-of-box problem is the light version of dist cache deployment: -libjars.

          Option #5) Add a MapReduce property that controls whether the MapReduce JARs are automatically shipped with the job via -libjars, and turn it on by default. Yes, it's inefficient (in both time and space), but it works out of the box and is a natural segue into dist cache deployment.

          Show
          templedf Daniel Templeton added a comment - As if we're not split enough on this decision, let me throw in one more option. The final and correct answer is that users should set up distributed cache deployment (per https://hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html ). At that point things just work, and upgrades become simpler. The problem is that distributed cache deployment isn't an option for out of the box. Spark does not have the problem we're trying to solve here. Every time you submit a Spark application, they ship the assembly JAR via the distributed cache. If we want to push users to distributed cache deployment, maybe the way to solve the out-of-box problem is the light version of dist cache deployment: -libjars. Option #5) Add a MapReduce property that controls whether the MapReduce JARs are automatically shipped with the job via -libjars, and turn it on by default. Yes, it's inefficient (in both time and space), but it works out of the box and is a natural segue into dist cache deployment.
          Hide
          rkanter Robert Kanter added a comment -

          Either #3 (adding HADOOP_MAPRED_HOME to the classpath) or #5 (though maybe not actually using -libjars argument, but doing something equivalent) sound good to me.

          Show
          rkanter Robert Kanter added a comment - Either #3 (adding HADOOP_MAPRED_HOME to the classpath) or #5 (though maybe not actually using -libjars argument, but doing something equivalent) sound good to me.
          Hide
          bibinchundatt Bibin A Chundatt added a comment - - edited

          Thank you Daniel Templeton and Robert Kanter for inputs

          #3 since for upgrade case when we have running apps and havnt used distributed cache.
          Need to modify only one config during upgrade and everything works as old.
          Another case can happen when old clients try to submit apps too rt?

          client doesn't have the same paths as the NM hosts, it won't work.

          Daniel Templeton Thank you for bring this scenarios too had missed out.

          Will do document update and upload patch soon.

          Show
          bibinchundatt Bibin A Chundatt added a comment - - edited Thank you Daniel Templeton and Robert Kanter for inputs #3 since for upgrade case when we have running apps and havnt used distributed cache. Need to modify only one config during upgrade and everything works as old. Another case can happen when old clients try to submit apps too rt? client doesn't have the same paths as the NM hosts, it won't work. Daniel Templeton Thank you for bring this scenarios too had missed out. Will do document update and upload patch soon.
          Hide
          templedf Daniel Templeton added a comment -

          Since this is a rather core issue, I'd love to see some additional opinions here. Andrew Wang, Varun Saxena, Akira Ajisaka, Sangjin Lee, Sunil G, Naganarasimha G R, Vinod Kumar Vavilapalli, any thoughts? Should we take this to the mailing list for more visibility?

          Show
          templedf Daniel Templeton added a comment - Since this is a rather core issue, I'd love to see some additional opinions here. Andrew Wang , Varun Saxena , Akira Ajisaka , Sangjin Lee , Sunil G , Naganarasimha G R , Vinod Kumar Vavilapalli , any thoughts? Should we take this to the mailing list for more visibility?
          Hide
          sjlee0 Sangjin Lee added a comment -

          I am +1 for #3. There is the yuck factor of mixing up MR into YARN, but the issue is significant enough and the change small enough that I think we can make an exception here.

          We can move the discussion to the mailing list to expand the audience.

          Show
          sjlee0 Sangjin Lee added a comment - I am +1 for #3. There is the yuck factor of mixing up MR into YARN, but the issue is significant enough and the change small enough that I think we can make an exception here. We can move the discussion to the mailing list to expand the audience.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1 for #3.
          In MAPREDUCE-6702, I wrote a patch to document that we need to set HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME to mapreduce.admin.user.env and yarn.app.mapreduce.am.env, but HADOOP_MAPRED_HOME is not always as same as HADOOP_COMMON_HOME. I'm thinking it's better to set yarn.nodemanager.env-whitelist instead of setting these parameters.

          Show
          ajisakaa Akira Ajisaka added a comment - +1 for #3. In MAPREDUCE-6702 , I wrote a patch to document that we need to set HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME to mapreduce.admin.user.env and yarn.app.mapreduce.am.env, but HADOOP_MAPRED_HOME is not always as same as HADOOP_COMMON_HOME. I'm thinking it's better to set yarn.nodemanager.env-whitelist instead of setting these parameters.
          Hide
          templedf Daniel Templeton added a comment -

          Yep. I was planning to submit a JIRA this morning to make exactly that change.

          Show
          templedf Daniel Templeton added a comment - Yep. I was planning to submit a JIRA this morning to make exactly that change.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Thank you all . Attaching patch based on solution 3

          Show
          bibinchundatt Bibin A Chundatt added a comment - Thank you all . Attaching patch based on solution 3
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvninstall 8m 25s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvnsite 1m 2s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          11m 20s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837039/MAPREDUCE-6704.0002.patch
          JIRA Issue MAPREDUCE-6704
          Optional Tests asflicense mvnsite
          uname Linux 3866c2eb9814 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 69dd5fa
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6796/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 8m 25s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvnsite 1m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 11m 20s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837039/MAPREDUCE-6704.0002.patch JIRA Issue MAPREDUCE-6704 Optional Tests asflicense mvnsite uname Linux 3866c2eb9814 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 69dd5fa modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6796/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          The patch looks fine. I'm OK with this solution. After thinking it over, I don't think we need to take it to the mailing list. As long as we're just updating docs, this doesn't need to be a high profile issue.

          Anyone have any concerns about this patch as the solution to the out-of-the-box issue for MapReduce?

          Show
          templedf Daniel Templeton added a comment - The patch looks fine. I'm OK with this solution. After thinking it over, I don't think we need to take it to the mailing list. As long as we're just updating docs, this doesn't need to be a high profile issue. Anyone have any concerns about this patch as the solution to the out-of-the-box issue for MapReduce?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Thank you Daniel Templeton for looking in to issue .
          Attaching html files for reference.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Thank you Daniel Templeton for looking in to issue . Attaching html files for reference.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvninstall 7m 45s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvnsite 0m 48s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          10m 19s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837039/MAPREDUCE-6704.0002.patch
          JIRA Issue MAPREDUCE-6704
          Optional Tests asflicense mvnsite
          uname Linux 011ef3f101a1 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d8bab3d
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6797/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 7m 45s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvnsite 0m 48s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 10m 19s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837039/MAPREDUCE-6704.0002.patch JIRA Issue MAPREDUCE-6704 Optional Tests asflicense mvnsite uname Linux 011ef3f101a1 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d8bab3d modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6797/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Sorry for coming in late.

          +1 for Option#3. I think its better if we whitelist MR related env variables.
          However do we need to whitelist JAVA_HOME by default?

          Show
          sunilg Sunil G added a comment - Sorry for coming in late. +1 for Option#3. I think its better if we whitelist MR related env variables. However do we need to whitelist JAVA_HOME by default?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          However do we need to whitelist JAVA_HOME by default?

          Its already part of the default list

          Show
          bibinchundatt Bibin A Chundatt added a comment - However do we need to whitelist JAVA_HOME by default? Its already part of the default list
          Hide
          sunilg Sunil G added a comment -

          Yes. Thanks for clarifying. My bad.

          Show
          sunilg Sunil G added a comment - Yes. Thanks for clarifying. My bad.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Bibin A Chundatt for providing the patch and html files. LGTM, +1.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Bibin A Chundatt for providing the patch and html files. LGTM, +1.
          Hide
          tangzhankun Zhankun Tang added a comment -

          Bibin A Chundatt, Daniel Templeton. Does option#3 works for dockerized MR application? I tried but failed(traditional MR works with option#3). I used "sequenceiq/hadoop-docker" Docker image. The image works in branch-2.8. Any idea?

          Show
          tangzhankun Zhankun Tang added a comment - Bibin A Chundatt , Daniel Templeton . Does option#3 works for dockerized MR application? I tried but failed(traditional MR works with option#3). I used "sequenceiq/hadoop-docker" Docker image. The image works in branch-2.8. Any idea?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Zhankun Tang

          I used "sequenceiq/hadoop-docker" Docker image. The image works in branch-2.8. Any idea?

          Sorry,No idea. Check with hadoop mailing list with logs

          Daniel Templeton
          Any more changes required for this jira

          Show
          bibinchundatt Bibin A Chundatt added a comment - Zhankun Tang I used "sequenceiq/hadoop-docker" Docker image. The image works in branch-2.8. Any idea? Sorry,No idea. Check with hadoop mailing list with logs Daniel Templeton Any more changes required for this jira
          Hide
          templedf Daniel Templeton added a comment -

          Zhankun Tang, I'm pretty surprised by that. The Docker container should inherit the env vars set for the container, which is what this whitelist property does. Could you run a Docker job with delayed cleanup (yarn.nodemanager.delete.debug-delay-sec) set so that you can see the launch script? That should give you a good idea what's going on.

          Bibin A Chundatt, I'm fine with the patch, but I would like to understand why it's not working for Docker. As there are alternative solutions, we should make sure we pick one that works across the board.

          Show
          templedf Daniel Templeton added a comment - Zhankun Tang , I'm pretty surprised by that. The Docker container should inherit the env vars set for the container, which is what this whitelist property does. Could you run a Docker job with delayed cleanup ( yarn.nodemanager.delete.debug-delay-sec ) set so that you can see the launch script? That should give you a good idea what's going on. Bibin A Chundatt , I'm fine with the patch, but I would like to understand why it's not working for Docker. As there are alternative solutions, we should make sure we pick one that works across the board.
          Hide
          bibinchundatt Bibin A Chundatt added a comment - - edited

          Daniel Templeton
          I was thinking of handling through mailing list and analyze the logs . We will other any other env needs to specified

          Zhankun Tang
          As Daniel mentioned after configuring yarn.nodemanager.delete.debug-delay-sec please attach the following

          1. launch script
          2. node manager logs during docker run.
          Show
          bibinchundatt Bibin A Chundatt added a comment - - edited Daniel Templeton I was thinking of handling through mailing list and analyze the logs . We will other any other env needs to specified Zhankun Tang As Daniel mentioned after configuring yarn.nodemanager.delete.debug-delay-sec please attach the following launch script node manager logs during docker run.
          Hide
          tangzhankun Zhankun Tang added a comment -

          Daniel Templeton, Bibin A Chundatt I have done testing again on branch2.8 and trunk to double-check this. The result is that trunk fails on Dockerized MRv2. I would like to upload the tarballs including container logs, NM logs and launch_container.sh.
          But I cannot find the button to upload them in this page while I can in YARN JIRAs. Is there any permission required?

          Show
          tangzhankun Zhankun Tang added a comment - Daniel Templeton , Bibin A Chundatt I have done testing again on branch2.8 and trunk to double-check this. The result is that trunk fails on Dockerized MRv2. I would like to upload the tarballs including container logs, NM logs and launch_container.sh. But I cannot find the button to upload them in this page while I can in YARN JIRAs. Is there any permission required?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Added Zhankun Tang into contributor role in MR JIRA. Now you can attach files here. Thanks.

          Show
          ajisakaa Akira Ajisaka added a comment - Added Zhankun Tang into contributor role in MR JIRA. Now you can attach files here. Thanks.
          Hide
          tangzhankun Zhankun Tang added a comment -

          The tarballs uploaded. Please check.

          Show
          tangzhankun Zhankun Tang added a comment - The tarballs uploaded. Please check.
          Hide
          tangzhankun Zhankun Tang added a comment -

          Thanks, Akira.

          Show
          tangzhankun Zhankun Tang added a comment - Thanks, Akira.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Zhankun Tang
          Thank you for attaching logs. Had quick look are the launch_container.sh
          HADOOP_MAPRED_HOME patch is set based nodemanager which should have worked in all cases.As per my understanding could fail only when the HADOOP_MAPRED_HOME where the container do launch is different.

          Could you try based on temp.patch .Currently i don't have docker setup to test my analysis.

          +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          @@ -320,11 +320,11 @@ public void writeLaunchEnv(OutputStream out, Map<String, String> environment,
                   ContainerLaunch.ShellScriptBuilder.create();
               Set<String> whitelist = new HashSet<>();
           
          -    whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
          -    whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
          -    whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
          -    whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
          -    whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
          +    String[] nmWhiteList = conf.get(YarnConfiguration.NM_ENV_WHITELIST,
          +        YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
          +    for (String param : nmWhiteList) {
          +      whitelist.add(param);
          +    }
          
          

          Daniel Templeton ContainerExecutor#writeLaunchEnv is handling only the default whitelist params as per current code.Looks like bug in yarn.
          The above change should allow taking the HADOOP_MAPRED_HOME from docker env during container launch.

          launcher_container.sh mapred_home_patch will get exported as following.

          export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-"/home/hadoopbuild"}
          

          instead of

          export HADOOP_MAPRED_HOME="/home/hadoopbuild"
          
          Show
          bibinchundatt Bibin A Chundatt added a comment - Zhankun Tang Thank you for attaching logs. Had quick look are the launch_container.sh HADOOP_MAPRED_HOME patch is set based nodemanager which should have worked in all cases.As per my understanding could fail only when the HADOOP_MAPRED_HOME where the container do launch is different. Could you try based on temp.patch .Currently i don't have docker setup to test my analysis. +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java @@ -320,11 +320,11 @@ public void writeLaunchEnv(OutputStream out, Map< String , String > environment, ContainerLaunch.ShellScriptBuilder.create(); Set< String > whitelist = new HashSet<>(); - whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name()); - whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name()); - whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name()); - whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name()); - whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name()); + String [] nmWhiteList = conf.get(YarnConfiguration.NM_ENV_WHITELIST, + YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split( "," ); + for ( String param : nmWhiteList) { + whitelist.add(param); + } Daniel Templeton ContainerExecutor#writeLaunchEnv is handling only the default whitelist params as per current code.Looks like bug in yarn. The above change should allow taking the HADOOP_MAPRED_HOME from docker env during container launch. launcher_container.sh mapred_home_patch will get exported as following. export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-"/home/hadoopbuild"} instead of export HADOOP_MAPRED_HOME="/home/hadoopbuild"
          Hide
          tangzhankun Zhankun Tang added a comment -

          Thanks. Bibin A Chundatt, I agree with your analysis. And will test your patch and give feedback asap.

          Show
          tangzhankun Zhankun Tang added a comment - Thanks. Bibin A Chundatt , I agree with your analysis. And will test your patch and give feedback asap.
          Hide
          tangzhankun Zhankun Tang added a comment - - edited

          Bibin A Chundatt, The temp.patch corrects the HADOOP_MAPRED_HOME variable and the MRAppMaster was found successfully. But then the AM fails on token version check. Please check the container logs uploaded(MR-6704-trunk-tempPatch.tar.gz).

          2016-11-14 04:43:47,660 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
          java.io.IOException: Exception reading /tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1479115665737_0004/container_1479115665737_0004_02_000001/container_tokens
                  at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:198)
                  at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:816)
                  at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
                  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
                  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1495)
                  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
          Caused by: java.io.IOException: Unknown version 1 in token storage.
                  at org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:216)
                  at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:195)
                  ... 5 more
          2016-11-14 04:43:47,662 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1
          
          Show
          tangzhankun Zhankun Tang added a comment - - edited Bibin A Chundatt , The temp.patch corrects the HADOOP_MAPRED_HOME variable and the MRAppMaster was found successfully. But then the AM fails on token version check. Please check the container logs uploaded(MR-6704-trunk-tempPatch.tar.gz). 2016-11-14 04:43:47,660 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.io.IOException: Exception reading /tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1479115665737_0004/container_1479115665737_0004_02_000001/container_tokens at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:198) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:816) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1495) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448) Caused by: java.io.IOException: Unknown version 1 in token storage. at org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:216) at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:195) ... 5 more 2016-11-14 04:43:47,662 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Not able to read token files could be rights related also. Could you check rights of nm-local-dir is 755.If its a test setup try removing nm-local-dir/usercache/yarn dir and run again.

          Daniel Templeton
          Once you look into the temp.patch will file a jira in yarn

          Show
          bibinchundatt Bibin A Chundatt added a comment - Not able to read token files could be rights related also. Could you check rights of nm-local-dir is 755.If its a test setup try removing nm-local-dir/usercache/yarn dir and run again. Daniel Templeton Once you look into the temp.patch will file a jira in yarn
          Hide
          tangzhankun Zhankun Tang added a comment -

          The rights of nm-local-dir is 755 and it can run MRv2 application. I tried removing nm-local-dir/usercache/yarn and ran dockerized MR, still same error.

          Show
          tangzhankun Zhankun Tang added a comment - The rights of nm-local-dir is 755 and it can run MRv2 application. I tried removing nm-local-dir/usercache/yarn and ran dockerized MR, still same error.
          Hide
          bibinchundatt Bibin A Chundatt added a comment - - edited

          Zhankun Tang could be related docker mount , YARN-5765 or 2.7 to hadoop.3.0 compatability Sorry i am new to docker so unaware
          Daniel Templeton Added YARN-5877 to track white list case.

          Show
          bibinchundatt Bibin A Chundatt added a comment - - edited Zhankun Tang could be related docker mount , YARN-5765 or 2.7 to hadoop.3.0 compatability Sorry i am new to docker so unaware Daniel Templeton Added YARN-5877 to track white list case.
          Hide
          vvasudev Varun Vasudev added a comment -

          The token exception is likely due to running a branch-2 MR tarball on trunk. The credential file format has changed from trunk to branch-2.

          Show
          vvasudev Varun Vasudev added a comment - The token exception is likely due to running a branch-2 MR tarball on trunk. The credential file format has changed from trunk to branch-2.
          Hide
          tangzhankun Zhankun Tang added a comment -

          Varun Vasudev, Yes, probably. The version of hadoop in Docker image is 2.7.1. But shouldn't the credential file format be backward-compatible?

          Show
          tangzhankun Zhankun Tang added a comment - Varun Vasudev , Yes, probably. The version of hadoop in Docker image is 2.7.1. But shouldn't the credential file format be backward-compatible?
          Hide
          aw Allen Wittenauer added a comment -

          But shouldn't the credential file format be backward-compatible?

          There is no guarantee that running hadoop 2.x jars will work carte blanche on a hadoop 3.x system. This is covered in lots of spots by the Apache Hadoop Compatibility guidelines. More to the point, the token file format is technically Private while the APIs are Public/Evolving. It's wide open to get changed incompatibility.

          Slightly off-topic: The token file format in pre-3.x is raw Java objects written to disk. This causes lots of issues, some obvious and some not-obvious. It is now protobuf which fixes a good chunk/all? of the issues. [I'll omit the list of problems that I'm aware of--which would be incomplete at best--due to length. It really is that broken. haha.]

          Show
          aw Allen Wittenauer added a comment - But shouldn't the credential file format be backward-compatible? There is no guarantee that running hadoop 2.x jars will work carte blanche on a hadoop 3.x system. This is covered in lots of spots by the Apache Hadoop Compatibility guidelines. More to the point, the token file format is technically Private while the APIs are Public/Evolving. It's wide open to get changed incompatibility. Slightly off-topic: The token file format in pre-3.x is raw Java objects written to disk. This causes lots of issues, some obvious and some not-obvious. It is now protobuf which fixes a good chunk/all? of the issues. [I'll omit the list of problems that I'm aware of--which would be incomplete at best--due to length. It really is that broken. haha.]
          Hide
          templedf Daniel Templeton added a comment -

          Zhankun Tang, you'll want to test with a branch-2 cluster so that you have all the latest fixes, so you'll either need to update the Docker image or create your own. Let us know how the testing goes.

          Show
          templedf Daniel Templeton added a comment - Zhankun Tang , you'll want to test with a branch-2 cluster so that you have all the latest fixes, so you'll either need to update the Docker image or create your own. Let us know how the testing goes.
          Hide
          tangzhankun Zhankun Tang added a comment -

          Daniel Templeton, Varun Vasudev, Bibin A Chundatt I have built a Docker image with hadoop-3.0.0-alpha2 in it. And it passed the testing. So the documentation and temp.patch submitted is needed for fixing this blocker.

          Show
          tangzhankun Zhankun Tang added a comment - Daniel Templeton , Varun Vasudev , Bibin A Chundatt I have built a Docker image with hadoop-3.0.0-alpha2 in it. And it passed the testing. So the documentation and temp.patch submitted is needed for fixing this blocker.
          Hide
          templedf Daniel Templeton added a comment -

          Thanks, Zhankun Tang. Just to be clear, you're saying that after resolving your versioning issues, the procedure documented in this patch worked for you?

          Show
          templedf Daniel Templeton added a comment - Thanks, Zhankun Tang . Just to be clear, you're saying that after resolving your versioning issues, the procedure documented in this patch worked for you?
          Hide
          tangzhankun Zhankun Tang added a comment -

          Daniel Templeton, Yes. First, the whitelist environment "yarn.nodemanager.env-whitelist " solved the traditional MRv2 failure. And the "temp.patch" solved the MRv2 class not found issue in launch_script.sh for LCE Docker support.

          Then to run a Dockerized MRv2 application in hadoop-3, we have to build a hadoop-3.0 Docker image to bypass the secure token version issue found in hadoop-2 Docker image currently. I think this also need to be documented.

          Show
          tangzhankun Zhankun Tang added a comment - Daniel Templeton , Yes. First, the whitelist environment "yarn.nodemanager.env-whitelist " solved the traditional MRv2 failure. And the "temp.patch" solved the MRv2 class not found issue in launch_script.sh for LCE Docker support. Then to run a Dockerized MRv2 application in hadoop-3, we have to build a hadoop-3.0 Docker image to bypass the secure token version issue found in hadoop-2 Docker image currently. I think this also need to be documented.
          Hide
          bibinchundatt Bibin A Chundatt added a comment - - edited

          Zhankun Tang thank you for providing test result. YARN-5877 is added to track the same.

          Daniel Templeton As per discussion we had identified that Solution 3 is best fit. But for docker to run by default we should support change YARN-5877 (Currently the vars to be fetched from env for docker is fixed we should make it configurable) .Comment explains in details why we should support.

          So combination of both would fix completely.

          Show
          bibinchundatt Bibin A Chundatt added a comment - - edited Zhankun Tang thank you for providing test result. YARN-5877 is added to track the same. Daniel Templeton As per discussion we had identified that Solution 3 is best fit. But for docker to run by default we should support change YARN-5877 (Currently the vars to be fetched from env for docker is fixed we should make it configurable) . Comment explains in details why we should support. So combination of both would fix completely.
          Hide
          templedf Daniel Templeton added a comment -

          Works for me.

          Show
          templedf Daniel Templeton added a comment - Works for me.
          Hide
          andrew.wang Andrew Wang added a comment -

          Ping, we getting close on resolving this JIRA?

          Show
          andrew.wang Andrew Wang added a comment - Ping, we getting close on resolving this JIRA?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Andrew Wang
          Yes we are getting close to resolving.Only documentation update is required as part of this jira.
          MAPREDUCE-6704.0002 should get committed after YARN-5877 is resolved.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Andrew Wang Yes we are getting close to resolving.Only documentation update is required as part of this jira. MAPREDUCE-6704 .0002 should get committed after YARN-5877 is resolved.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching same patch again

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching same patch again
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvninstall 14m 20s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 mvnsite 0m 58s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          17m 15s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843551/MAPREDUCE-6704.0002.patch
          JIRA Issue MAPREDUCE-6704
          Optional Tests asflicense mvnsite
          uname Linux a919cfe90daa 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cee0c46
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6846/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 14m 20s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvnsite 0m 58s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 17m 15s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843551/MAPREDUCE-6704.0002.patch JIRA Issue MAPREDUCE-6704 Optional Tests asflicense mvnsite uname Linux a919cfe90daa 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cee0c46 modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6846/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1 again.

          Show
          ajisakaa Akira Ajisaka added a comment - +1 again.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed. Thanks all!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed. Thanks all!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11031 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11031/)
          MAPREDUCE-6704. Update the documents to run MapReduce application. (aajisaka: rev 22befbd585f65934e1d9ae5782a8f961192c0750)

          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/SingleCluster.md.vm
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11031 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11031/ ) MAPREDUCE-6704 . Update the documents to run MapReduce application. (aajisaka: rev 22befbd585f65934e1d9ae5782a8f961192c0750) (edit) hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md (edit) hadoop-common-project/hadoop-common/src/site/markdown/SingleCluster.md.vm

            People

            • Assignee:
              bibinchundatt Bibin A Chundatt
              Reporter:
              bibinchundatt Bibin A Chundatt
            • Votes:
              1 Vote for this issue
              Watchers:
              29 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development