Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6204

TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    1. MAPREDUCE-6204-4.patch
      1 kB
      sam liu
    2. MAPREDUCE-6204-3.patch
      1 kB
      sam liu
    3. MAPREDUCE-6204-2.patch
      1 kB
      sam liu
    4. MAPREDUCE-6204-1.patch
      1 kB
      sam liu
    5. MAPREDUCE-6204.patch
      1 kB
      sam liu

      Activity

      Hide
      ozawa Tsuyoshi Ozawa added a comment -

      OK, thank you

      Show
      ozawa Tsuyoshi Ozawa added a comment - OK, thank you
      Hide
      jira.shegalov Gera Shegalov added a comment -

      Tsuyoshi, my thinking is it's a neutral change: it did not make the project better, reverting won't make it better either. It's just an overhead. Don't revert

      Show
      jira.shegalov Gera Shegalov added a comment - Tsuyoshi, my thinking is it's a neutral change: it did not make the project better, reverting won't make it better either. It's just an overhead. Don't revert
      Hide
      ozawa Tsuyoshi Ozawa added a comment -

      Gera Shegalov Yes, I agree with you. Let me claryfy - I commited this change not because of avoiding environment problem, but because of updating to use new property instead of deprecated one. Does this make sense? If this desicion is wrong, I have option to revert this change. Please ping me in the case.

      Show
      ozawa Tsuyoshi Ozawa added a comment - Gera Shegalov Yes, I agree with you. Let me claryfy - I commited this change not because of avoiding environment problem, but because of updating to use new property instead of deprecated one. Does this make sense? If this desicion is wrong, I have option to revert this change. Please ping me in the case.
      Hide
      jira.shegalov Gera Shegalov added a comment -

      Hi Tsuyoshi Ozawa, I am sorry for coming in late after a break, but I think we should avoid making extra commits to fix obvious environment problems.

      Show
      jira.shegalov Gera Shegalov added a comment - Hi Tsuyoshi Ozawa , I am sorry for coming in late after a break, but I think we should avoid making extra commits to fix obvious environment problems.
      Hide
      hudson Hudson added a comment -

      SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2152 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2152/)
      MAPREDUCE-6204. TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a)

      • hadoop-mapreduce-project/CHANGES.txt
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
      Show
      hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2152 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2152/ ) MAPREDUCE-6204 . TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
      Hide
      hudson Hudson added a comment -

      SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #204 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/204/)
      MAPREDUCE-6204. TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a)

      • hadoop-mapreduce-project/CHANGES.txt
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
      Show
      hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #204 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/204/ ) MAPREDUCE-6204 . TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
      Hide
      hudson Hudson added a comment -

      FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #194 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/194/)
      MAPREDUCE-6204. TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a)

      • hadoop-mapreduce-project/CHANGES.txt
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
      Show
      hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #194 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/194/ ) MAPREDUCE-6204 . TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
      Hide
      hudson Hudson added a comment -

      FAILURE: Integrated in Hadoop-Hdfs-trunk #2134 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2134/)
      MAPREDUCE-6204. TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a)

      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
      • hadoop-mapreduce-project/CHANGES.txt
      Show
      hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2134 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2134/ ) MAPREDUCE-6204 . TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java hadoop-mapreduce-project/CHANGES.txt
      Hide
      hudson Hudson added a comment -

      SUCCESS: Integrated in Hadoop-Yarn-trunk #936 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/936/)
      MAPREDUCE-6204. TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a)

      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
      • hadoop-mapreduce-project/CHANGES.txt
      Show
      hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #936 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/936/ ) MAPREDUCE-6204 . TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java hadoop-mapreduce-project/CHANGES.txt
      Hide
      hudson Hudson added a comment -

      SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #205 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/205/)
      MAPREDUCE-6204. TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a)

      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
      • hadoop-mapreduce-project/CHANGES.txt
      Show
      hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #205 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/205/ ) MAPREDUCE-6204 . TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java hadoop-mapreduce-project/CHANGES.txt
      Hide
      hudson Hudson added a comment -

      FAILURE: Integrated in Hadoop-trunk-Commit #7894 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7894/)
      MAPREDUCE-6204. TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a)

      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
      • hadoop-mapreduce-project/CHANGES.txt
      Show
      hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7894 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7894/ ) MAPREDUCE-6204 . TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. (ozawa: rev b27ce3c5318a40bc79565783961f6675dcdcb14a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java hadoop-mapreduce-project/CHANGES.txt
      Hide
      ozawa Tsuyoshi Ozawa added a comment -

      Committed this to trunk and branch-2. Thanks sam liu for your contribution and thanks Gera Shegalov for your review!

      Show
      ozawa Tsuyoshi Ozawa added a comment - Committed this to trunk and branch-2. Thanks sam liu for your contribution and thanks Gera Shegalov for your review!
      Hide
      hadoopqa Hadoop QA added a comment -



      +1 overall



      Vote Subsystem Runtime Comment
      0 pre-patch 5m 29s Pre-patch trunk compilation is healthy.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
      +1 javac 7m 38s There were no new javac warning messages.
      +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
      +1 whitespace 0m 0s The patch has no lines that end in whitespace.
      +1 install 1m 37s mvn install still works.
      +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
      +1 mapreduce tests 104m 7s Tests passed in hadoop-mapreduce-client-jobclient.
          119m 46s  



      Subsystem Report/Notes
      Patch URL http://issues.apache.org/jira/secure/attachment/12732465/MAPREDUCE-6204-4.patch
      Optional Tests javac unit
      git revision trunk / ab7958f
      hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5751/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
      Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5751/testReport/
      Java 1.7.0_55
      uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5751/console

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 29s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 38s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 mapreduce tests 104m 7s Tests passed in hadoop-mapreduce-client-jobclient.     119m 46s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732465/MAPREDUCE-6204-4.patch Optional Tests javac unit git revision trunk / ab7958f hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5751/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5751/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5751/console This message was automatically generated.
      Hide
      ozawa Tsuyoshi Ozawa added a comment -

      Before committing, I'd like to run tests. Kicking Jenkins.

      Show
      ozawa Tsuyoshi Ozawa added a comment - Before committing, I'd like to run tests. Kicking Jenkins.
      Hide
      ozawa Tsuyoshi Ozawa added a comment -

      +1, committing this shortly.

      Show
      ozawa Tsuyoshi Ozawa added a comment - +1, committing this shortly.
      Hide
      ozawa Tsuyoshi Ozawa added a comment -

      OK. I agree with the fix for using newer properties instead of using deprecated one.

      Gera Shegalov As you mentioned, the test failure can be not related and it's addressed on MAPREDUCE-6205. However, we should move using newer, we can say more proper, properties. Do you agree with fixing this?

      Show
      ozawa Tsuyoshi Ozawa added a comment - OK. I agree with the fix for using newer properties instead of using deprecated one. Gera Shegalov As you mentioned, the test failure can be not related and it's addressed on MAPREDUCE-6205 . However, we should move using newer, we can say more proper, properties. Do you agree with fixing this?
      Hide
      sam liu sam liu added a comment -

      Hi Tsuyoshi,

      In fact, there is no hadoop cluster and hadoop related configuration in my dev env on power linux, so I also did not sure why the 'MAPRED_MAP_TASK_JAVA_OPTS' and 'MAPRED_REDUCE_TASK_JAVA_OPTS' has above default and unexpected value there. At the same time, we mentioned that the root causes of the issue is described in MAPREDUCE-6205, however its patch is still not reviewed/accepted yet. Therefore, current fix will greatly make sense to the ut TestJobCounters: it explicitly replaces the deprecated property with the legal ones, and really could set correct value to MAP/REDUCE OPTS properties and fix the unexpected env/configurations issue and make the ut more robust.

      Thanks!

      Show
      sam liu sam liu added a comment - Hi Tsuyoshi, In fact, there is no hadoop cluster and hadoop related configuration in my dev env on power linux, so I also did not sure why the 'MAPRED_MAP_TASK_JAVA_OPTS' and 'MAPRED_REDUCE_TASK_JAVA_OPTS' has above default and unexpected value there. At the same time, we mentioned that the root causes of the issue is described in MAPREDUCE-6205 , however its patch is still not reviewed/accepted yet. Therefore, current fix will greatly make sense to the ut TestJobCounters: it explicitly replaces the deprecated property with the legal ones, and really could set correct value to MAP/REDUCE OPTS properties and fix the unexpected env/configurations issue and make the ut more robust. Thanks!
      Hide
      ozawa Tsuyoshi Ozawa added a comment -

      sam liu, thank you for pinging us. Let me clarify one point: the configurations as follows looks set by your configurations as Gera Shegalov mentioned. Is it right?

      -Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
      
      Show
      ozawa Tsuyoshi Ozawa added a comment - sam liu , thank you for pinging us. Let me clarify one point: the configurations as follows looks set by your configurations as Gera Shegalov mentioned. Is it right? -Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/ var /hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/ var /hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/ var /hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
      Hide
      sam liu sam liu added a comment -

      It's really weird that there always be some ut failures, however none of them was brought by current fix and all the ut could pass in my env. I guess there might be some env issues.

      I suggest to rerun the ut or deliver this fix directly after reviewing.

      Thanks!

      Show
      sam liu sam liu added a comment - It's really weird that there always be some ut failures, however none of them was brought by current fix and all the ut could pass in my env. I guess there might be some env issues. I suggest to rerun the ut or deliver this fix directly after reviewing. Thanks!
      Hide
      hadoopqa Hadoop QA added a comment -



      -1 overall



      Vote Subsystem Runtime Comment
      0 pre-patch 5m 22s Pre-patch trunk compilation is healthy.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
      +1 javac 7m 40s There were no new javac warning messages.
      +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
      +1 whitespace 0m 0s The patch has no lines that end in whitespace.
      +1 install 1m 33s mvn install still works.
      +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
      -1 mapreduce tests 89m 27s Tests failed in hadoop-mapreduce-client-jobclient.
          104m 55s  



      Reason Tests
      Failed unit tests hadoop.mapreduce.security.ssl.TestEncryptedShuffle
        hadoop.mapred.TestLineRecordReaderJobs
        hadoop.mapreduce.security.TestBinaryTokenFile
        hadoop.mapred.TestCombineSequenceFileInputFormat
        hadoop.mapreduce.TestMapReduceLazyOutput
        hadoop.mapreduce.TestMRJobClient
      Timed out tests org.apache.hadoop.mapreduce.TestLargeSort



      Subsystem Report/Notes
      Patch URL http://issues.apache.org/jira/secure/attachment/12732465/MAPREDUCE-6204-4.patch
      Optional Tests javac unit
      git revision trunk / 2463666
      hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5721/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
      Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5721/testReport/
      Java 1.7.0_55
      uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5721/console

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 22s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 40s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. -1 mapreduce tests 89m 27s Tests failed in hadoop-mapreduce-client-jobclient.     104m 55s   Reason Tests Failed unit tests hadoop.mapreduce.security.ssl.TestEncryptedShuffle   hadoop.mapred.TestLineRecordReaderJobs   hadoop.mapreduce.security.TestBinaryTokenFile   hadoop.mapred.TestCombineSequenceFileInputFormat   hadoop.mapreduce.TestMapReduceLazyOutput   hadoop.mapreduce.TestMRJobClient Timed out tests org.apache.hadoop.mapreduce.TestLargeSort Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732465/MAPREDUCE-6204-4.patch Optional Tests javac unit git revision trunk / 2463666 hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5721/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5721/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5721/console This message was automatically generated.
      Hide
      hadoopqa Hadoop QA added a comment -



      -1 overall



      Vote Subsystem Runtime Comment
      0 pre-patch 5m 11s Pre-patch trunk compilation is healthy.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
      +1 javac 7m 31s There were no new javac warning messages.
      +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
      +1 whitespace 0m 0s The patch has no lines that end in whitespace.
      +1 install 1m 33s mvn install still works.
      +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
      -1 mapreduce tests 99m 0s Tests failed in hadoop-mapreduce-client-jobclient.
          114m 11s  



      Reason Tests
      Failed unit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler



      Subsystem Report/Notes
      Patch URL http://issues.apache.org/jira/secure/attachment/12732201/MAPREDUCE-6204-3.patch
      Optional Tests javac unit
      git revision trunk / 360dff5
      hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5712/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
      Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5712/testReport/
      Java 1.7.0_55
      uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5712/console

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 11s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 mapreduce tests 99m 0s Tests failed in hadoop-mapreduce-client-jobclient.     114m 11s   Reason Tests Failed unit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732201/MAPREDUCE-6204-3.patch Optional Tests javac unit git revision trunk / 360dff5 hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5712/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5712/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5712/console This message was automatically generated.
      Hide
      sam liu sam liu added a comment -

      These ut should not fail, as they could pass in my env.

      I studied the outputs of ut execution and seems the failures were caused by env issue, like 'java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/ApplicationIdNotProvidedException'.

      Could please help rerun the ut?

      Thanks!

      Show
      sam liu sam liu added a comment - These ut should not fail, as they could pass in my env. I studied the outputs of ut execution and seems the failures were caused by env issue, like 'java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/ApplicationIdNotProvidedException'. Could please help rerun the ut? Thanks!
      Hide
      hadoopqa Hadoop QA added a comment -



      -1 overall



      Vote Subsystem Runtime Comment
      0 pre-patch 5m 15s Pre-patch trunk compilation is healthy.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
      +1 javac 7m 32s There were no new javac warning messages.
      +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
      +1 whitespace 0m 0s The patch has no lines that end in whitespace.
      +1 install 1m 36s mvn install still works.
      +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
      -1 mapreduce tests 94m 49s Tests failed in hadoop-mapreduce-client-jobclient.
          110m 12s  



      Reason Tests
      Failed unit tests hadoop.mapred.TestJobCounters
        hadoop.mapred.TestJobSysDirWithDFS
        hadoop.mapred.TestReduceFetchFromPartialMem



      Subsystem Report/Notes
      Patch URL http://issues.apache.org/jira/secure/attachment/12730785/MAPREDUCE-6204-2.patch
      Optional Tests javac unit
      git revision trunk / a583a40
      hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5650/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
      Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5650/testReport/
      Java 1.7.0_55
      uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5650/console

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 15s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. -1 mapreduce tests 94m 49s Tests failed in hadoop-mapreduce-client-jobclient.     110m 12s   Reason Tests Failed unit tests hadoop.mapred.TestJobCounters   hadoop.mapred.TestJobSysDirWithDFS   hadoop.mapred.TestReduceFetchFromPartialMem Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730785/MAPREDUCE-6204-2.patch Optional Tests javac unit git revision trunk / a583a40 hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5650/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5650/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5650/console This message was automatically generated.
      Hide
      sam liu sam liu added a comment -

      Created a new patch basing on latest trunk code(2015/05/06)

      Show
      sam liu sam liu added a comment - Created a new patch basing on latest trunk code(2015/05/06)
      Hide
      hadoopqa Hadoop QA added a comment -

      The patch artifact directory has been removed!
      This is a fatal error for test-patch.sh. Aborting.
      Jenkins (node H3) information at https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5629/ may provide some hints.

      Show
      hadoopqa Hadoop QA added a comment - The patch artifact directory has been removed! This is a fatal error for test-patch.sh. Aborting. Jenkins (node H3) information at https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5629/ may provide some hints.
      Hide
      hadoopqa Hadoop QA added a comment -



      -1 overall



      Vote Subsystem Runtime Comment
      0 pre-patch 5m 5s Pre-patch trunk compilation is healthy.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
      +1 javac 7m 27s There were no new javac warning messages.
      +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
      +1 checkstyle 0m 38s There were no new checkstyle issues.
      -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
      +1 install 1m 34s mvn install still works.
      +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
      +1 findbugs 0m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
      +1 mapreduce tests 104m 31s Tests passed in hadoop-mapreduce-client-jobclient.
          120m 51s  



      Subsystem Report/Notes
      Patch URL http://issues.apache.org/jira/secure/attachment/12688804/MAPREDUCE-6204-1.patch
      Optional Tests javac unit findbugs checkstyle
      git revision trunk / a319771
      whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5624/artifact/patchprocess/whitespace.txt
      hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5624/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
      Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5624/testReport/
      Java 1.7.0_55
      uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5624/console

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 27s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 38s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 0m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 104m 31s Tests passed in hadoop-mapreduce-client-jobclient.     120m 51s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12688804/MAPREDUCE-6204-1.patch Optional Tests javac unit findbugs checkstyle git revision trunk / a319771 whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5624/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5624/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5624/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5624/console This message was automatically generated.
      Hide
      sam liu sam liu added a comment -

      Gera, thanks for your comments and I will update the patch of MAPREDUCE-6205 according to your suggestion later.

      Show
      sam liu sam liu added a comment - Gera, thanks for your comments and I will update the patch of MAPREDUCE-6205 according to your suggestion later.
      Hide
      jira.shegalov Gera Shegalov added a comment -

      I added following code to get the property value during map/reduce task execution in TestJobCounters#MemoryLoaderMapper#configure() ...

      These properties are actually meant to be used by the MRAppMaster. So although they are accessible in task attempt JVM's, it's not really correct to verify them there when they play no role.

      Show
      jira.shegalov Gera Shegalov added a comment - I added following code to get the property value during map/reduce task execution in TestJobCounters#MemoryLoaderMapper#configure() ... These properties are actually meant to be used by the MRAppMaster. So although they are accessible in task attempt JVM's, it's not really correct to verify them there when they play no role.
      Hide
      jira.shegalov Gera Shegalov added a comment -

      You are correct that mapred.child.java.opts should be added to the deprecation deltas. Let us continue working on this in MAPREDUCE-6205.

      However, this is not the root cause for the failure you are seeing on your ppc64. You are just working around the issue that you somewhere have this config that definitely does not belong to a clean build:

      -Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
      
      Show
      jira.shegalov Gera Shegalov added a comment - You are correct that mapred.child.java.opts should be added to the deprecation deltas. Let us continue working on this in MAPREDUCE-6205 . However, this is not the root cause for the failure you are seeing on your ppc64. You are just working around the issue that you somewhere have this config that definitely does not belong to a clean build: -Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/ var /hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/ var /hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/ var /hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
      Hide
      sam liu sam liu added a comment -

      For many deprecated properties, hadoop code always set its value to its new version of property. Absolutely, this solution is clear and won't bring any misunderstanding on the conception of 'deprecation'.

      However, for the deprecated property "mapred.child.java.opts", hadoop uses a different method: it won't update the value of its new versions MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. Yes, finally hadoop will set a correct value on the classpath, however the program like MapReduce could not get correct value of MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS: it definately bring confusing to the users.

      Furthermore, this ut will fail without this patch on ppc64 platform, but could pass on the same env with this patch. Actually the root cause is the insuffcient completion of the deprecation of property "mapred.child.java.opts". We could not say this patch is useless or simply pass the buck to 'env issue': if this is an env issue, this ut should fail on the ppc64 platform either with the deprecated property "mapred.child.java.opts" or with the new version of properties MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS.

      Therefore this patch is necessary and could remedy the insuffcient completion of the deprecation of property "mapred.child.java.opts".

      Show
      sam liu sam liu added a comment - For many deprecated properties, hadoop code always set its value to its new version of property. Absolutely, this solution is clear and won't bring any misunderstanding on the conception of 'deprecation'. However, for the deprecated property "mapred.child.java.opts", hadoop uses a different method: it won't update the value of its new versions MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. Yes, finally hadoop will set a correct value on the classpath, however the program like MapReduce could not get correct value of MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS: it definately bring confusing to the users. Furthermore, this ut will fail without this patch on ppc64 platform, but could pass on the same env with this patch. Actually the root cause is the insuffcient completion of the deprecation of property "mapred.child.java.opts". We could not say this patch is useless or simply pass the buck to 'env issue': if this is an env issue, this ut should fail on the ppc64 platform either with the deprecated property "mapred.child.java.opts" or with the new version of properties MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. Therefore this patch is necessary and could remedy the insuffcient completion of the deprecation of property "mapred.child.java.opts".
      Hide
      jira.shegalov Gera Shegalov added a comment -

      +0. The patch is not wrong but it has zero effect on a clean environment. In contrast, clearly, in sam liu's ppc64 environment there is a config file on the classpath defining non-default map.java.opts.

      Show
      jira.shegalov Gera Shegalov added a comment - +0. The patch is not wrong but it has zero effect on a clean environment. In contrast, clearly, in sam liu 's ppc64 environment there is a config file on the classpath defining non-default map.java.opts.
      Hide
      ozawa Tsuyoshi Ozawa added a comment -

      I'll commit latest patch in this weekend if there is no opposition.

      Show
      ozawa Tsuyoshi Ozawa added a comment - I'll commit latest patch in this weekend if there is no opposition.
      Hide
      ozawa Tsuyoshi Ozawa added a comment -

      Sam, thanks for your clarification.

      +1 against MAPREDUCE-6204-1.patch with following reasons:
      1. We should use new properties instead of deprecated one.
      2. On some environment, JVM options are configured wrongly. As a result, TestJobCounters can fails. The patch fix the problem.

      Gera Shegalov please let me know if you have opinions.

      Show
      ozawa Tsuyoshi Ozawa added a comment - Sam, thanks for your clarification. +1 against MAPREDUCE-6204 -1.patch with following reasons: 1. We should use new properties instead of deprecated one. 2. On some environment, JVM options are configured wrongly. As a result, TestJobCounters can fails. The patch fix the problem. Gera Shegalov please let me know if you have opinions.
      Hide
      sam liu sam liu added a comment -

      Hi Tsuyoshi,

      I did more tests on PPC64, it has some difference from x86:
      1. Without any change, the map/reduce tasks will print:
      1.1 on x86:
      MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G
      MAPRED_MAP_TASK_JAVA_OPTS==null
      MAPRED_REDUCE_TASK_JAVA_OPTS=null

      1.2 on PPC64:
      MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G
      MAPRED_MAP_TASK_JAVA_OPTS=-Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
      MAPRED_REDUCE_TASK_JAVA_OPTS=-Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd

      2. With this patch, both x86 and PPC64 will print same message:
      MAPRED_TASK_JAVA_OPTS=-Xmx200m
      MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G
      MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G

      I will look into PPC64 part to see if there is any other issue there.

      I also opened another jira MAPREDUCE-6205 to update MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS with MAPRED_TASK_JAVA_OPTS together, but the patch brought another new ut failure. I also will look into that jira later.

      So this ut really need this patch on PPC64 now and as you said we should use new properties instead of deprecated one.

      Thanks!

      Show
      sam liu sam liu added a comment - Hi Tsuyoshi, I did more tests on PPC64, it has some difference from x86: 1. Without any change, the map/reduce tasks will print: 1.1 on x86: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS==null MAPRED_REDUCE_TASK_JAVA_OPTS=null 1.2 on PPC64: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=-Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd MAPRED_REDUCE_TASK_JAVA_OPTS=-Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd 2. With this patch, both x86 and PPC64 will print same message: MAPRED_TASK_JAVA_OPTS=-Xmx200m MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G I will look into PPC64 part to see if there is any other issue there. I also opened another jira MAPREDUCE-6205 to update MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS with MAPRED_TASK_JAVA_OPTS together, but the patch brought another new ut failure. I also will look into that jira later. So this ut really need this patch on PPC64 now and as you said we should use new properties instead of deprecated one. Thanks!
      Hide
      ozawa Tsuyoshi Ozawa added a comment -

      Basically, +1 for the fix since we should use new properties instead of deprecated one.

      My question, maybe Gera also has, is why the test could pass on PPC64 with same JVM options? I'd like to clarify it since it can be another bug if you can see JVM crash without your patch.

      Show
      ozawa Tsuyoshi Ozawa added a comment - Basically, +1 for the fix since we should use new properties instead of deprecated one. My question, maybe Gera also has, is why the test could pass on PPC64 with same JVM options? I'd like to clarify it since it can be another bug if you can see JVM crash without your patch.
      Hide
      sam liu sam liu added a comment -

      Tsuyoshi,

      Yes, you are correct and the two properties won't be updated.

      I did a test:
      1. In TestTaskAttemptContainerRequest.java, I added 'jobConf.set(JobConf.MAPRED_TASK_JAVA_OPTS, "-Xms32m -Xmx1G")'
      2. Then I found the command args created by 'ContainerLaunchContext launchCtx = TaskAttemptImpl.createContainerLaunchContext...' likes:
      =$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xms32m -Xmx1G -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 127.0.0.1 0 attempt_1_0001_m_000001_1 0 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr

      Please mention that only a commn heap setting "-Xms32m -Xmx1G" is added into the command args, but the two properties(MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS) were not updated as expected.

      So I think it's still necessary to update MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS values in TestJobCounters, as that test could pass on PPC64 only with such updates.

      Show
      sam liu sam liu added a comment - Tsuyoshi, Yes, you are correct and the two properties won't be updated. I did a test: 1. In TestTaskAttemptContainerRequest.java, I added 'jobConf.set(JobConf.MAPRED_TASK_JAVA_OPTS, "-Xms32m -Xmx1G")' 2. Then I found the command args created by 'ContainerLaunchContext launchCtx = TaskAttemptImpl.createContainerLaunchContext...' likes: =$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xms32m -Xmx1G -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 127.0.0.1 0 attempt_1_0001_m_000001_1 0 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr Please mention that only a commn heap setting "-Xms32m -Xmx1G" is added into the command args, but the two properties(MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS) were not updated as expected. So I think it's still necessary to update MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS values in TestJobCounters, as that test could pass on PPC64 only with such updates.
      Hide
      ozawa Tsuyoshi Ozawa added a comment -

      Sam,
      Note that JobConf.MAPRED_MAP_TASK_JAVA_OPTS/JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS are not updated when MapReduceChildJVM#getChildJavaOpts is called. Please check the method:

          if (isMapTask) {
            userClasspath = 
                jobConf.get(
                    JobConf.MAPRED_MAP_TASK_JAVA_OPTS, 
                    jobConf.get(
                        JobConf.MAPRED_TASK_JAVA_OPTS, 
                        JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS)
                );
           ...
          } else {
            userClasspath =
                jobConf.get(
                    JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS, 
                    jobConf.get(
                        JobConf.MAPRED_TASK_JAVA_OPTS,
                        JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS)
                    );
             ...
          }
      
      Show
      ozawa Tsuyoshi Ozawa added a comment - Sam, Note that JobConf.MAPRED_MAP_TASK_JAVA_OPTS/JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS are not updated when MapReduceChildJVM#getChildJavaOpts is called. Please check the method: if (isMapTask) { userClasspath = jobConf.get( JobConf.MAPRED_MAP_TASK_JAVA_OPTS, jobConf.get( JobConf.MAPRED_TASK_JAVA_OPTS, JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS) ); ... } else { userClasspath = jobConf.get( JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS, jobConf.get( JobConf.MAPRED_TASK_JAVA_OPTS, JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS) ); ... }
      Hide
      sam liu sam liu added a comment -

      Gera,

      I see, but could you please help explain why the map/reduce tasks could not get expected value of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS, but only return 'null' in the test?

      I added following code to get the property value during map/reduce task execution in TestJobCounters#MemoryLoaderMapper#configure() and TestJobCounters#MemoryLoaderReducer#configure():
      System.out.println("MAPRED_TASK_JAVA_OPTS=" + conf.get(JobConf.MAPRED_TASK_JAVA_OPTS));
      System.out.println("MAPRED_MAP_TASK_JAVA_OPTS=" + conf.get(JobConf.MAPRED_MAP_TASK_JAVA_OPTS));
      System.out.println("MAPRED_REDUCE_TASK_JAVA_OPTS=" + conf.get(JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS));

      Show
      sam liu sam liu added a comment - Gera, I see, but could you please help explain why the map/reduce tasks could not get expected value of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS, but only return 'null' in the test? I added following code to get the property value during map/reduce task execution in TestJobCounters#MemoryLoaderMapper#configure() and TestJobCounters#MemoryLoaderReducer#configure(): System.out.println("MAPRED_TASK_JAVA_OPTS=" + conf.get(JobConf.MAPRED_TASK_JAVA_OPTS)); System.out.println("MAPRED_MAP_TASK_JAVA_OPTS=" + conf.get(JobConf.MAPRED_MAP_TASK_JAVA_OPTS)); System.out.println("MAPRED_REDUCE_TASK_JAVA_OPTS=" + conf.get(JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS));
      Hide
      jira.shegalov Gera Shegalov added a comment -

      getVMCommand is invoked to build the launch context in o.a.h.mapreduce.v2.app.job.impl.TaskAttemptImpl#createContainerLaunchContext. If you modify the test to set the DEBUG log level for MRAppMaster you will even find the launch context printed in its syslog.

      In TestJobCounters#runHeapUsageTestJob add

      jobConf.set(MRJobConfig.MR_AM_LOG_LEVEL, "DEBUG");
      
      Show
      jira.shegalov Gera Shegalov added a comment - getVMCommand is invoked to build the launch context in o.a.h.mapreduce.v2.app.job.impl.TaskAttemptImpl#createContainerLaunchContext . If you modify the test to set the DEBUG log level for MRAppMaster you will even find the launch context printed in its syslog. In TestJobCounters#runHeapUsageTestJob add jobConf.set(MRJobConfig.MR_AM_LOG_LEVEL, "DEBUG" );
      Hide
      sam liu sam liu added a comment -

      I did many tests and it does not work if only making MAPRED_TASK_JAVA_OPTS larger heap size.

      Actually, MapReduceChildJVM#getChildJavaOpts is only be invoked by method MapReduceChildJVM#getVMCommand which is not be invoked by any other existing hadoop code: that's why in my tests the value of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS is still null and not be the same as MAPRED_TASK_JAVA_OPTS. Only I set correct value to MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS, the map/reduce tasks could get correct java opts as below and this unit test could pass on PPC64 platform:
      MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G
      MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G
      MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G

      Show
      sam liu sam liu added a comment - I did many tests and it does not work if only making MAPRED_TASK_JAVA_OPTS larger heap size. Actually, MapReduceChildJVM#getChildJavaOpts is only be invoked by method MapReduceChildJVM#getVMCommand which is not be invoked by any other existing hadoop code: that's why in my tests the value of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS is still null and not be the same as MAPRED_TASK_JAVA_OPTS. Only I set correct value to MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS, the map/reduce tasks could get correct java opts as below and this unit test could pass on PPC64 platform: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G
      Hide
      ozawa Tsuyoshi Ozawa added a comment -

      I checked MapReduceChildJVM#getChildJavaOpts as Gera mentioned. The method includes following code:

          if (isMapTask) {
            userClasspath = 
                jobConf.get(
                    JobConf.MAPRED_MAP_TASK_JAVA_OPTS, 
                    jobConf.get(
                        JobConf.MAPRED_TASK_JAVA_OPTS, 
                        JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS)
                );
           ...
          } else {
            userClasspath =
                jobConf.get(
                    JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS, 
                    jobConf.get(
                        JobConf.MAPRED_TASK_JAVA_OPTS,
                        JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS)
                    );
             ...
          }
      

      It means that MAPRED_TASK_JAVA_OPTS is used instead of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS if MAPRED_TASK_JAVA_OPTS is set.

      MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G
      MAPRED_MAP_TASK_JAVA_OPTS=null
      MAPRED_REDUCE_TASK_JAVA_OPTS=null

      This looks correct behavior since MapReduceChildJVM#getChildJavaOpts just gets the value of MAPRED_TASK_JAVA_OPTS instead of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS. I think the patch doesn't fix the problem. What do you think? How about making MAPRED_TASK_JAVA_OPTS larger heap size?

      Show
      ozawa Tsuyoshi Ozawa added a comment - I checked MapReduceChildJVM#getChildJavaOpts as Gera mentioned. The method includes following code: if (isMapTask) { userClasspath = jobConf.get( JobConf.MAPRED_MAP_TASK_JAVA_OPTS, jobConf.get( JobConf.MAPRED_TASK_JAVA_OPTS, JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS) ); ... } else { userClasspath = jobConf.get( JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS, jobConf.get( JobConf.MAPRED_TASK_JAVA_OPTS, JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS) ); ... } It means that MAPRED_TASK_JAVA_OPTS is used instead of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS if MAPRED_TASK_JAVA_OPTS is set. MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=null MAPRED_REDUCE_TASK_JAVA_OPTS=null This looks correct behavior since MapReduceChildJVM#getChildJavaOpts just gets the value of MAPRED_TASK_JAVA_OPTS instead of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS. I think the patch doesn't fix the problem. What do you think? How about making MAPRED_TASK_JAVA_OPTS larger heap size?
      Hide
      hadoopqa Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12688804/MAPREDUCE-6204-1.patch
      against trunk revision 5caebba.

      +1 @author. The patch does not contain any @author tags.

      +1 tests included. The patch appears to include 1 new or modified test files.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 javadoc. There were no new javadoc warning messages.

      +1 eclipse:eclipse. The patch built with eclipse:eclipse.

      +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

      Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5089//testReport/
      Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5089//console

      This message is automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12688804/MAPREDUCE-6204-1.patch against trunk revision 5caebba. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5089//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5089//console This message is automatically generated.
      Hide
      sam liu sam liu added a comment -

      Hi Tsuyoshi,

      I attached new patch MAPREDUCE-6204-1.patch and could you please help take a review?

      Thanks!

      Show
      sam liu sam liu added a comment - Hi Tsuyoshi, I attached new patch MAPREDUCE-6204 -1.patch and could you please help take a review? Thanks!
      Hide
      sam liu sam liu added a comment -

      Attached new patch. Thanks!

      Show
      sam liu sam liu added a comment - Attached new patch. Thanks!
      Hide
      sam liu sam liu added a comment -

      Gera,

      I did a test:
      1. Without the patch, current test TestJobCounters#MemoryLoaderMapper and MemoryLoaderReducer will get value 'null' for properties MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS during MR job execution:
      MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G
      MAPRED_MAP_TASK_JAVA_OPTS=null
      MAPRED_REDUCE_TASK_JAVA_OPTS=null

      2. With the patch, current test TestJobCounters#MemoryLoaderMapper and MemoryLoaderReducer will get correct value for all the properties:
      MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G
      MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G
      MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G

      Show
      sam liu sam liu added a comment - Gera, I did a test: 1. Without the patch, current test TestJobCounters#MemoryLoaderMapper and MemoryLoaderReducer will get value 'null' for properties MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS during MR job execution: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=null MAPRED_REDUCE_TASK_JAVA_OPTS=null 2. With the patch, current test TestJobCounters#MemoryLoaderMapper and MemoryLoaderReducer will get correct value for all the properties: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G
      Hide
      jira.shegalov Gera Shegalov added a comment -

      I am not sure it helps with crashes because memory settings have not changed with this patch if you look at
      MapReduceChildJVM#getChildJavaOpts

      Show
      jira.shegalov Gera Shegalov added a comment - I am not sure it helps with crashes because memory settings have not changed with this patch if you look at MapReduceChildJVM#getChildJavaOpts
      Hide
      ozawa Tsuyoshi Ozawa added a comment -

      sam liu, the fix itself good to me, but the patch is not valid one. Could you generate the patch by using git diff --no-prefix command?

      Show
      ozawa Tsuyoshi Ozawa added a comment - sam liu , the fix itself good to me, but the patch is not valid one. Could you generate the patch by using git diff --no-prefix command?
      Hide
      hadoopqa Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12688787/MAPREDUCE-6204.patch
      against trunk revision fdf042d.

      -1 patch. The patch command could not apply the patch.

      Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5088//console

      This message is automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12688787/MAPREDUCE-6204.patch against trunk revision fdf042d. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5088//console This message is automatically generated.
      Hide
      sam liu sam liu added a comment -

      The solution is to replace the deprecated property MAPRED_TASK_JAVA_OPTS with its new values MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS in the test.

      After applying the patch, current test could pass on PPC64 platform all the time.

      Show
      sam liu sam liu added a comment - The solution is to replace the deprecated property MAPRED_TASK_JAVA_OPTS with its new values MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS in the test. After applying the patch, current test could pass on PPC64 platform all the time.
      Hide
      sam liu sam liu added a comment -

      In Hadoop 2.x, the property JobConf.MAPRED_TASK_JAVA_OPTS is deprecated, and hadoop will ignore this property directly without replacing its new values(MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS).

      This issue will cause test TestJobCounters fail on PPC64 platform sometimes, as PPC64 JVM sometimes comsumes more memory when executing map/reduce tasks and might encounter JVM crashing issue once the default heap size is exhausted.

      Show
      sam liu sam liu added a comment - In Hadoop 2.x, the property JobConf.MAPRED_TASK_JAVA_OPTS is deprecated, and hadoop will ignore this property directly without replacing its new values(MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS). This issue will cause test TestJobCounters fail on PPC64 platform sometimes, as PPC64 JVM sometimes comsumes more memory when executing map/reduce tasks and might encounter JVM crashing issue once the default heap size is exhausted.

        People

        • Assignee:
          sam liu sam liu
          Reporter:
          sam liu sam liu
        • Votes:
          0 Vote for this issue
          Watchers:
          7 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development