Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      From https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt TestMRJobsWithProfiler fails intermittently

      Running org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
      Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 212.973 sec <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
      testDifferentProfilers(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler)  Time elapsed: 133.116 sec  <<< FAILURE!
      java.lang.AssertionError: expected:<4> but was:<1>
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.failNotEquals(Assert.java:743)
      	at org.junit.Assert.assertEquals(Assert.java:118)
      	at org.junit.Assert.assertEquals(Assert.java:555)
      	at org.junit.Assert.assertEquals(Assert.java:542)
      	at org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfilerInternal(TestMRJobsWithProfiler.java:212)
      	at org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testDifferentProfilers(TestMRJobsWithProfiler.java:117)
      

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9487 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9487/)
        MAPREDUCE-6580. Test failure: TestMRJobsWithProfiler. Contributed by (aajisaka: rev 0bfe5a0167f0af65f1d7a8bc251a1c404ed41316)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithProfiler.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9487 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9487/ ) MAPREDUCE-6580 . Test failure: TestMRJobsWithProfiler. Contributed by (aajisaka: rev 0bfe5a0167f0af65f1d7a8bc251a1c404ed41316) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithProfiler.java
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Committed this to branch-2.7 and above. Thanks Eric Badger for the contribution!

        Show
        ajisakaa Akira Ajisaka added a comment - Committed this to branch-2.7 and above. Thanks Eric Badger for the contribution!
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Nice catch, Eric! LGTM, +1.

        Show
        ajisakaa Akira Ajisaka added a comment - Nice catch, Eric! LGTM, +1.
        Hide
        ebadger Eric Badger added a comment -

        TestUberAM and TestMRJobs both have failed recent builds in MAPREDUCE-6315 and are unrelated to this patch.

        TestMRIntermediateDataEncryption fails intermittently with a timeout in recent builds MAPREDUCE-6579, YARN-4340.

        Show
        ebadger Eric Badger added a comment - TestUberAM and TestMRJobs both have failed recent builds in MAPREDUCE-6315 and are unrelated to this patch. TestMRIntermediateDataEncryption fails intermittently with a timeout in recent builds MAPREDUCE-6579 , YARN-4340 .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 56s trunk passed
        +1 compile 0m 19s trunk passed with JDK v1.8.0_74
        +1 compile 0m 22s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 28s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 29s trunk passed
        +1 javadoc 0m 11s trunk passed with JDK v1.8.0_74
        +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 17s the patch passed with JDK v1.8.0_74
        +1 javac 0m 17s the patch passed
        +1 compile 0m 19s the patch passed with JDK v1.7.0_95
        +1 javac 0m 19s the patch passed
        +1 checkstyle 0m 11s the patch passed
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 35s the patch passed
        +1 javadoc 0m 9s the patch passed with JDK v1.8.0_74
        +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95
        -1 unit 94m 51s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_74.
        -1 unit 110m 49s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95.
        -1 asflicense 0m 28s Patch generated 19 ASF License warnings.
        219m 22s



        Reason Tests
        JDK v1.8.0_74 Failed junit tests hadoop.mapreduce.v2.TestUberAM
          hadoop.mapreduce.v2.TestMRJobs
        JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.v2.TestUberAM
          hadoop.mapreduce.v2.TestMRJobs
        JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.mapred.TestMRIntermediateDataEncryption



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794629/MAPREDUCE-6580.001.patch
        JIRA Issue MAPREDUCE-6580
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux cf3665d3f430 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e7ed05e
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_74.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 56s trunk passed +1 compile 0m 19s trunk passed with JDK v1.8.0_74 +1 compile 0m 22s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 29s trunk passed +1 javadoc 0m 11s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 23s the patch passed +1 compile 0m 17s the patch passed with JDK v1.8.0_74 +1 javac 0m 17s the patch passed +1 compile 0m 19s the patch passed with JDK v1.7.0_95 +1 javac 0m 19s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 35s the patch passed +1 javadoc 0m 9s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95 -1 unit 94m 51s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_74. -1 unit 110m 49s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 28s Patch generated 19 ASF License warnings. 219m 22s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.mapreduce.v2.TestUberAM   hadoop.mapreduce.v2.TestMRJobs JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.v2.TestUberAM   hadoop.mapreduce.v2.TestMRJobs JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.mapred.TestMRIntermediateDataEncryption Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794629/MAPREDUCE-6580.001.patch JIRA Issue MAPREDUCE-6580 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cf3665d3f430 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e7ed05e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6388/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        I am able to get TestMRJobsWithProfiler#testDifferentProfilers to fail consistently on my Mac. However, I am not able to get it to fail on my Linux box or on either box when running testDefaultProfiler. On my mac, I have found that the problem is with the hprof option "times" in TestMRJobsWithProfiler.java:137 (-agentlib:hprof=cpu=times,heap=sites,force=n,thread=y,verbose=n," + "file=%s). This is causing the JVM to screw up and fail seemingly before it even begins to start the container.

        Additionally, the "times" option is incredibly expensive and slows the JVM down a ton, making the test very likely to timeout on a busy Jenkins instance given the 150 second test timeout. IMO using "samples" instead of "times" would be much better. Using samples, the test passes on my mac and speeds the test up by almost 2x (74.8s vs. 145.5s on my Linux machine). Using "samples" would get rid of the hprof error on my mac and also speed up the test considerably, reducing the chance of test timeouts.

        Is there any reason that we can't use "samples" instead of "times" for the profiler?

        Show
        ebadger Eric Badger added a comment - I am able to get TestMRJobsWithProfiler#testDifferentProfilers to fail consistently on my Mac. However, I am not able to get it to fail on my Linux box or on either box when running testDefaultProfiler. On my mac, I have found that the problem is with the hprof option "times" in TestMRJobsWithProfiler.java:137 (-agentlib:hprof=cpu=times,heap=sites,force=n,thread=y,verbose=n," + "file=%s). This is causing the JVM to screw up and fail seemingly before it even begins to start the container. Additionally, the "times" option is incredibly expensive and slows the JVM down a ton, making the test very likely to timeout on a busy Jenkins instance given the 150 second test timeout. IMO using "samples" instead of "times" would be much better. Using samples, the test passes on my mac and speeds the test up by almost 2x (74.8s vs. 145.5s on my Linux machine). Using "samples" would get rid of the hprof error on my mac and also speed up the test considerably, reducing the chance of test timeouts. Is there any reason that we can't use "samples" instead of "times" for the profiler?

          People

          • Assignee:
            ebadger Eric Badger
            Reporter:
            rohithsharma Rohith Sharma K S
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development