Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6223

TestJobConf#testNegativeValueForTaskVmem failures

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.328 sec <<< FAILURE! - in org.apache.hadoop.conf.TestJobConf
      testNegativeValueForTaskVmem(org.apache.hadoop.conf.TestJobConf)  Time elapsed: 0.089 sec  <<< FAILURE!
      java.lang.AssertionError: expected:<1024> but was:<-1>
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.failNotEquals(Assert.java:743)
      	at org.junit.Assert.assertEquals(Assert.java:118)
      	at org.junit.Assert.assertEquals(Assert.java:555)
      	at org.junit.Assert.assertEquals(Assert.java:542)
      	at org.apache.hadoop.conf.TestJobConf.testNegativeValueForTaskVmem(TestJobConf.java:111)
      
      1. MAPREDUCE-6223.006.patch
        5 kB
        Varun Saxena
      2. MAPREDUCE-6223.005.patch
        5 kB
        Varun Saxena
      3. MAPREDUCE-6223.004.patch
        5 kB
        Varun Saxena
      4. MAPREDUCE-6223.003.patch
        5 kB
        Varun Saxena
      5. MAPREDUCE-6223.002.patch
        5 kB
        Varun Saxena
      6. MAPREDUCE-6223.001.patch
        3 kB
        Varun Saxena

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Same question here as I just posed on MAPREDUCE-6234, do we need to mark this change as incompatible if it's only present with MAPREDUCE-5785, which is already marked incompatible and only in 3.0.0?

          Show
          andrew.wang Andrew Wang added a comment - Same question here as I just posed on MAPREDUCE-6234 , do we need to mark this change as incompatible if it's only present with MAPREDUCE-5785 , which is already marked incompatible and only in 3.0.0?
          Hide
          varun_saxena Varun Saxena added a comment -

          Thanks Karthik Kambatla/Tsuyoshi Ozawa for the review and commit

          Show
          varun_saxena Varun Saxena added a comment - Thanks Karthik Kambatla / Tsuyoshi Ozawa for the review and commit
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2067 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2067/)
          MAPREDUCE-6223. TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2067 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2067/ ) MAPREDUCE-6223 . TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #117 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/117/)
          MAPREDUCE-6223. TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #117 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/117/ ) MAPREDUCE-6223 . TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #108 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/108/)
          MAPREDUCE-6223. TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #108 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/108/ ) MAPREDUCE-6223 . TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2049 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2049/)
          MAPREDUCE-6223. TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2049 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2049/ ) MAPREDUCE-6223 . TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #851 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/851/)
          MAPREDUCE-6223. TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #851 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/851/ ) MAPREDUCE-6223 . TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #117 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/117/)
          MAPREDUCE-6223. TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #117 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/117/ ) MAPREDUCE-6223 . TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks Varun for working on this, and Tsuyoshi for help with reviews.

          Just committed to trunk.

          Show
          kasha Karthik Kambatla added a comment - Thanks Varun for working on this, and Tsuyoshi for help with reviews. Just committed to trunk.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7213 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7213/)
          MAPREDUCE-6223. TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7213 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7213/ ) MAPREDUCE-6223 . TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) (kasha: rev 1047c883ba01a252dbd4203e525fb1ff8ea313aa) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          Hide
          kasha Karthik Kambatla added a comment -

          Don't think the test failures are related. Latest patch looks good to me. +1. Will commit this shortly.

          Show
          kasha Karthik Kambatla added a comment - Don't think the test failures are related. Latest patch looks good to me. +1. Will commit this shortly.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12701152/MAPREDUCE-6223.006.patch
          against trunk revision dce8b9c.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
          org.apache.hadoop.mapred.TestMRTimelineEventHandling
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
          org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
          org.apache.hadoop.mapred.TestClusterMapReduceTestCase
          org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5227//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5227//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12701152/MAPREDUCE-6223.006.patch against trunk revision dce8b9c. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapred.TestMRTimelineEventHandling org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5227//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5227//console This message is automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          Uploading new patch after removing unnecessary log

          Show
          varun_saxena Varun Saxena added a comment - Uploading new patch after removing unnecessary log
          Hide
          varun_saxena Varun Saxena added a comment -

          Oh. This is actually a leftover log.
          Will remove it and upload a new patch

          Show
          varun_saxena Varun Saxena added a comment - Oh. This is actually a leftover log. Will remove it and upload a new patch
          Hide
          kasha Karthik Kambatla added a comment -

          Sorry for the delay in following up, and bringing up another nit:

          We should probably not log the memory used by MR task at info level. Also, it could use a little more description than "Value is"

          Show
          kasha Karthik Kambatla added a comment - Sorry for the delay in following up, and bringing up another nit: We should probably not log the memory used by MR task at info level. Also, it could use a little more description than "Value is"
          Hide
          varun_saxena Varun Saxena added a comment -

          Tsuyoshi Ozawa / Karthik Kambatla, kindly review the changes

          Show
          varun_saxena Varun Saxena added a comment - Tsuyoshi Ozawa / Karthik Kambatla , kindly review the changes
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12699116/MAPREDUCE-6223.005.patch
          against trunk revision 447bd7b.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5199//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5199//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5199//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12699116/MAPREDUCE-6223.005.patch against trunk revision 447bd7b. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5199//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5199//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5199//console This message is automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          Tsuyoshi Ozawa, ok will update these cases AND upload a new patch

          Show
          varun_saxena Varun Saxena added a comment - Tsuyoshi Ozawa , ok will update these cases AND upload a new patch
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Varun Saxena I think Karthik meant the lines as follows in addition to the lines as you mentioned:

          -    configuration.set(MRJobConfig.MAP_MEMORY_MB, String.valueOf(-1));
          +    configuration.set(MRJobConfig.MAP_MEMORY_MB, String.valueOf(100));
          
          -    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, String.valueOf(-1));
          +    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, String.valueOf(200));
          

          Would you update the case to include these cases?

          Show
          ozawa Tsuyoshi Ozawa added a comment - Varun Saxena I think Karthik meant the lines as follows in addition to the lines as you mentioned: - configuration.set(MRJobConfig.MAP_MEMORY_MB, String .valueOf(-1)); + configuration.set(MRJobConfig.MAP_MEMORY_MB, String .valueOf(100)); - configuration.set(MRJobConfig.REDUCE_MEMORY_MB, String .valueOf(-1)); + configuration.set(MRJobConfig.REDUCE_MEMORY_MB, String .valueOf(200)); Would you update the case to include these cases?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12697859/MAPREDUCE-6223.004.patch
          against trunk revision 3f5431a.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapred.TestClusterMapReduceTestCase

          The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapred.TestMRTimelineEventHandling
          org.apache.hadoop.mapred.TestMRIntermediateDataEncryption

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5180//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5180//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5180//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697859/MAPREDUCE-6223.004.patch against trunk revision 3f5431a. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestClusterMapReduceTestCase The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestMRTimelineEventHandling org.apache.hadoop.mapred.TestMRIntermediateDataEncryption Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5180//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5180//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5180//console This message is automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          Oops, that came out really bad. I meant the test below.

          	
          @Test	
          public void testNegativeValuesForMemoryParams() {
            JobConf configuration = new JobConf();
            configuration.set(MRJobConfig.MAP_MEMORY_MB, "-5");
            configuration.set(MRJobConfig.REDUCE_MEMORY_MB, "-6");
            Assert.assertEquals(-5, configuration.getMemoryForMapTask());
            Assert.assertEquals(-6, configuration.getMemoryForReduceTask());
          }
          
          Show
          varun_saxena Varun Saxena added a comment - Oops, that came out really bad. I meant the test below. @Test public void testNegativeValuesForMemoryParams() { JobConf configuration = new JobConf(); configuration.set(MRJobConfig.MAP_MEMORY_MB, "-5" ); configuration.set(MRJobConfig.REDUCE_MEMORY_MB, "-6" ); Assert.assertEquals(-5, configuration.getMemoryForMapTask()); Assert.assertEquals(-6, configuration.getMemoryForReduceTask()); }
          Hide
          varun_saxena Varun Saxena added a comment -

          Karthik Kambatla, you mean this test ?

          
          

          /**

          • Test that negative values for new configuration keys get passed through.
            */
            @Test
            public void testNegativeValuesForMemoryParams() {
            JobConf configuration = new JobConf();

          configuration.set(MRJobConfig.MAP_MEMORY_MB, "-5");
          configuration.set(MRJobConfig.REDUCE_MEMORY_MB, "-6");
          Assert.assertEquals(-5, configuration.getMemoryForMapTask());
          Assert.assertEquals(-6, configuration.getMemoryForReduceTask());
          }

          Show
          varun_saxena Varun Saxena added a comment - Karthik Kambatla , you mean this test ? /** Test that negative values for new configuration keys get passed through. */ @Test public void testNegativeValuesForMemoryParams() { JobConf configuration = new JobConf(); configuration.set(MRJobConfig.MAP_MEMORY_MB, "-5"); configuration.set(MRJobConfig.REDUCE_MEMORY_MB, "-6"); Assert.assertEquals(-5, configuration.getMemoryForMapTask()); Assert.assertEquals(-6, configuration.getMemoryForReduceTask()); }
          Hide
          kasha Karthik Kambatla added a comment -

          Patch looks mostly good to me. Nit: I would leave the test for negative values, but update the asserts to reflect the expected behavior.

          Show
          kasha Karthik Kambatla added a comment - Patch looks mostly good to me. Nit: I would leave the test for negative values, but update the asserts to reflect the expected behavior.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Varun Saxena for updating the patch. +1 pending Karthik Kambatla's review. The findbugs warnings look unrelated to the patch.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Varun Saxena for updating the patch. +1 pending Karthik Kambatla 's review. The findbugs warnings look unrelated to the patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12697264/MAPREDUCE-6223.003.patch
          against trunk revision cfb829e.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5176//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5176//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5176//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697264/MAPREDUCE-6223.003.patch against trunk revision cfb829e. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5176//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5176//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5176//console This message is automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          Akira Ajisaka, updated a new patch

          Show
          varun_saxena Varun Saxena added a comment - Akira Ajisaka , updated a new patch
          Hide
          kasha Karthik Kambatla added a comment -

          I would like to look at the patch before it is committed.

          Show
          kasha Karthik Kambatla added a comment - I would like to look at the patch before it is committed.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Varun Saxena and Masatake Iwasaki for the comments! I understand this issue should be fixed first.
          The v2 patch looks good to me.

          -    Assert.assertEquals(configuration.getMemoryForReduceTask(),2);
          -    
          +    Assert.assertEquals(configuration.getMemoryForReduceTask(),2);    
          

          Would you please remove the trailing the whitespaces? I'm +1 after that is addressed.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Varun Saxena and Masatake Iwasaki for the comments! I understand this issue should be fixed first. The v2 patch looks good to me. - Assert.assertEquals(configuration.getMemoryForReduceTask(),2); - + Assert.assertEquals(configuration.getMemoryForReduceTask(),2); Would you please remove the trailing the whitespaces? I'm +1 after that is addressed.
          Hide
          varun_saxena Varun Saxena added a comment -

          Yeah getting it from another constant should be fine as well.

          Show
          varun_saxena Varun Saxena added a comment - Yeah getting it from another constant should be fine as well.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          s/not local value but//

          Show
          iwasakims Masatake Iwasaki added a comment - s/not local value but//
          Hide
          iwasakims Masatake Iwasaki added a comment -

          I think JobConf#getMemoryReuiqred should get 1024 from not local value but constant in MRJobConfig other than DEFAULT_*_MEMORY_MB because 1024 is never set in Configuration. Akira Ajisaka / Tsuyoshi Ozawa, please commit the patch of this issue first. I will update the patch of MAPREDUCE-6234 later.

          Show
          iwasakims Masatake Iwasaki added a comment - I think JobConf#getMemoryReuiqred should get 1024 from not local value but constant in MRJobConfig other than DEFAULT_*_MEMORY_MB because 1024 is never set in Configuration. Akira Ajisaka / Tsuyoshi Ozawa , please commit the patch of this issue first. I will update the patch of MAPREDUCE-6234 later.
          Hide
          varun_saxena Varun Saxena added a comment -

          Akira Ajisaka / Tsuyoshi Ozawa,
          As you wish. Because currently test failures are appearing till MAPREDUCE-6223 is committed.
          Ideally MRJobConfig.DEFAULT_MAP_MEMORY_MB should not be changed. I feel we should not be taking default value from a local variable.
          MAPREDUCE-6234 hence will be a redundant fix as we will have to revert its changes again.

          Although it is somewhat confusing that default value in mapred-default.xml is -1 and in code we take it as 1024. But if somebody reads the config description, which should be done, its quite clear what is the behavior of this config.

          <description>
            The amount of memory to request from the scheduler for each 		
            map task. If this is not specified or is non-positive, it is inferred from
            mapreduce.map.java.opts and mapreduce.job.heap.memory-mb.ratio.
            If java-opts are also not specified, we set it to 1024.
          </description>
          

          You can take a call whether to commit that or not. Alternatively you can review and commit this as well.

          Show
          varun_saxena Varun Saxena added a comment - Akira Ajisaka / Tsuyoshi Ozawa , As you wish. Because currently test failures are appearing till MAPREDUCE-6223 is committed. Ideally MRJobConfig.DEFAULT_MAP_MEMORY_MB should not be changed. I feel we should not be taking default value from a local variable. MAPREDUCE-6234 hence will be a redundant fix as we will have to revert its changes again. Although it is somewhat confusing that default value in mapred-default.xml is -1 and in code we take it as 1024. But if somebody reads the config description, which should be done, its quite clear what is the behavior of this config. <description> The amount of memory to request from the scheduler for each map task. If this is not specified or is non-positive, it is inferred from mapreduce.map.java.opts and mapreduce.job.heap.memory-mb.ratio. If java-opts are also not specified, we set it to 1024. </description> You can take a call whether to commit that or not. Alternatively you can review and commit this as well.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Varun Saxena and Gera Shegalov, would you please take a look at MAPREDUCE-6234? The patch there will fix the failure.
          Can I commit the patch first and then we improve the tests by this issue and MAPREDUCE-6225?

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Varun Saxena and Gera Shegalov , would you please take a look at MAPREDUCE-6234 ? The patch there will fix the failure. Can I commit the patch first and then we improve the tests by this issue and MAPREDUCE-6225 ?
          Hide
          varun_saxena Varun Saxena added a comment -

          Findbugs as mentioned earlier will be addressed by MAPREDUCE-6225 and test failure is unrelated

          Show
          varun_saxena Varun Saxena added a comment - Findbugs as mentioned earlier will be addressed by MAPREDUCE-6225 and test failure is unrelated
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696034/MAPREDUCE-6223.002.patch
          against trunk revision 8acc5e9.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.TestLargeSort

          The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5143//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5143//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5143//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696034/MAPREDUCE-6223.002.patch against trunk revision 8acc5e9. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.TestLargeSort The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5143//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5143//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5143//console This message is automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          Thanks for your comments, Gera Shegalov.

          I think the test captured an overlooked case. Thus, instead of fixing the test code we should fix JobConf#getMemoryForMapTask and JobConf#getMemoryForReduceTask

          Agree

          Raised MAPREDUCE-6239 for merging of these 2 classes

          Show
          varun_saxena Varun Saxena added a comment - Thanks for your comments, Gera Shegalov . I think the test captured an overlooked case. Thus, instead of fixing the test code we should fix JobConf#getMemoryForMapTask and JobConf#getMemoryForReduceTask Agree Raised MAPREDUCE-6239 for merging of these 2 classes
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Thanks for taking this JIRA, Varun Saxena!

          Regarding merging the 2 test classes, it makes sense to consolidate both TestJobConf but let us do it in a separate JIRA.

          I think the test captured an overlooked case. Thus, instead of fixing the test code we should fix JobConf#getMemoryForMapTask and JobConf#getMemoryForReduceTask

          Show
          jira.shegalov Gera Shegalov added a comment - Thanks for taking this JIRA, Varun Saxena ! Regarding merging the 2 test classes, it makes sense to consolidate both TestJobConf but let us do it in a separate JIRA. I think the test captured an overlooked case. Thus, instead of fixing the test code we should fix JobConf#getMemoryForMapTask and JobConf#getMemoryForReduceTask
          Hide
          varun_saxena Varun Saxena added a comment -

          Findbugs to be tracked by MAPREDUCE-6225.

          Show
          varun_saxena Varun Saxena added a comment - Findbugs to be tracked by MAPREDUCE-6225 .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12694445/MAPREDUCE-6223.001.patch
          against trunk revision 35f6496.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.TestLargeSort

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5120//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5120//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5120//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12694445/MAPREDUCE-6223.001.patch against trunk revision 35f6496. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.TestLargeSort Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5120//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5120//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5120//console This message is automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          There is a TestJobConf class in both hadoop-mapreduce-client-jobclient and hadoop-mapreduce-client-core. Shouldnt they be merged ?

          Show
          varun_saxena Varun Saxena added a comment - There is a TestJobConf class in both hadoop-mapreduce-client-jobclient and hadoop-mapreduce-client-core . Shouldnt they be merged ?

            People

            • Assignee:
              varun_saxena Varun Saxena
              Reporter:
              jira.shegalov Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development