Hadoop Common
  1. Hadoop Common
  2. HADOOP-8851

Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.1-alpha
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: test
    • Labels:
      None

      Description

      This can help to reveal the cause of issue in the event of OOME in tests.
      Suggested patch attached.

      1. HADOOP-8851-vs-trunk.patch
        0.7 kB
        Ivan A. Veselovsky

        Activity

        Thomas Graves made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1214 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1214/)
        HADOOP-8851. Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests. Contributed by Ivan A. Veselovsky. (Revision 1392466)

        Result = FAILURE
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392466
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1214 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1214/ ) HADOOP-8851 . Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests. Contributed by Ivan A. Veselovsky. (Revision 1392466) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392466 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1183 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1183/)
        HADOOP-8851. Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests. Contributed by Ivan A. Veselovsky. (Revision 1392466)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392466
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1183 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1183/ ) HADOOP-8851 . Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests. Contributed by Ivan A. Veselovsky. (Revision 1392466) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392466 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #392 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/392/)
        svn merge -c 1392466 FIXES: HADOOP-8851. Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests. Contributed by Ivan A. Veselovsky. (Revision 1392521)

        Result = UNSTABLE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392521
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #392 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/392/ ) svn merge -c 1392466 FIXES: HADOOP-8851 . Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests. Contributed by Ivan A. Veselovsky. (Revision 1392521) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392521 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
        Robert Joseph Evans made changes -
        Fix Version/s 0.23.5 [ 12323314 ]
        Hide
        Robert Joseph Evans added a comment -

        I pulled this into branch-0.23 too

        Show
        Robert Joseph Evans added a comment - I pulled this into branch-0.23 too
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2820 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2820/)
        HADOOP-8851. Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests. Contributed by Ivan A. Veselovsky. (Revision 1392466)

        Result = FAILURE
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392466
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2820 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2820/ ) HADOOP-8851 . Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests. Contributed by Ivan A. Veselovsky. (Revision 1392466) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392466 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2798 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2798/)
        HADOOP-8851. Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests. Contributed by Ivan A. Veselovsky. (Revision 1392466)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392466
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2798 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2798/ ) HADOOP-8851 . Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests. Contributed by Ivan A. Veselovsky. (Revision 1392466) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392466 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2861 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2861/)
        HADOOP-8851. Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests. Contributed by Ivan A. Veselovsky. (Revision 1392466)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392466
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2861 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2861/ ) HADOOP-8851 . Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests. Contributed by Ivan A. Veselovsky. (Revision 1392466) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392466 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
        Aaron T. Myers made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 2.0.3-alpha [ 12323273 ]
        Resolution Fixed [ 1 ]
        Hide
        Aaron T. Myers added a comment -

        I've just committed this to trunk and branch-2.

        Thanks a lot for the contribution, Ivan.

        Show
        Aaron T. Myers added a comment - I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Ivan.
        Aaron T. Myers made changes -
        Affects Version/s 2.0.1-alpha [ 12322467 ]
        Target Version/s 2.0.3-alpha [ 12323273 ]
        Component/s test [ 12311440 ]
        Hide
        Aaron T. Myers added a comment -

        Thanks a lot for the info, Ivan, and for testing out the patch. I've also just tested out this option with OpenJDK and confirmed that it works similarly to the Sun JDK.

        I'm going to go ahead and commit this shortly.

        Show
        Aaron T. Myers added a comment - Thanks a lot for the info, Ivan, and for testing out the patch. I've also just tested out this option with OpenJDK and confirmed that it works similarly to the Sun JDK. I'm going to go ahead and commit this shortly.
        Hide
        Ivan A. Veselovsky added a comment -

        If you're using Jenkins (Hudson) builds, also it's good idea to save */.hprof artifacts to protect the memory dumps from being deleted upon the workspace cleanup.

        Show
        Ivan A. Veselovsky added a comment - If you're using Jenkins (Hudson) builds, also it's good idea to save * / .hprof artifacts to protect the memory dumps from being deleted upon the workspace cleanup.
        Hide
        Ivan A. Veselovsky added a comment -

        For example, add the following test:

        /**

        • Test to OOME: with the -XX:+HeapDumpOnOutOfMemoryError option the mem dump should be created by the JVM.
        • @throws Exception
          */
          public void testOOME() throws Exception
          Unknown macro: { final List<Object> list = new LinkedList<Object>(); while (true) { Object placeHolder = new HashMap<Object,Object>(); list.add(placeHolder); } }

        Typical output will look like the following:


        Running org.apache.hadoop.fs.permission.TestStickyBit
        java.lang.OutOfMemoryError: Java heap space
        Dumping heap to java_pid14838.hprof ...
        Heap dump file created [1515333529 bytes in 21.641 secs]
        Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 85.68 sec <<< FAILURE!


        The heap dump (this is a huge binary file) will be named java_pid<process_pid>.hprof, and will be created by JVM in the current directory of the test run process (e.g. .../hadoop-common/hadoop-hdfs-project/hadoop-hdfs/ in my case). The heap dump can be opened and investigated with almost any profiler, including NetBeans.

        Note, however, that the "-XX" ( http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html ) are HostSpot options, they may not work on other JVM implementations.
        But, afaik, most of the testing is done on Orecle's JVMs 1.6.0_XX, so the option will work and will be helpful in case of OOME problems.
        As experience shows, if there are no OOMEs, this option does not appear to introduce any problems or performance penalties.

        Show
        Ivan A. Veselovsky added a comment - For example, add the following test: /** Test to OOME: with the -XX:+HeapDumpOnOutOfMemoryError option the mem dump should be created by the JVM. @throws Exception */ public void testOOME() throws Exception Unknown macro: { final List<Object> list = new LinkedList<Object>(); while (true) { Object placeHolder = new HashMap<Object,Object>(); list.add(placeHolder); } } Typical output will look like the following: Running org.apache.hadoop.fs.permission.TestStickyBit java.lang.OutOfMemoryError: Java heap space Dumping heap to java_pid14838.hprof ... Heap dump file created [1515333529 bytes in 21.641 secs] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 85.68 sec <<< FAILURE! The heap dump (this is a huge binary file) will be named java_pid<process_pid>.hprof, and will be created by JVM in the current directory of the test run process (e.g. .../hadoop-common/hadoop-hdfs-project/hadoop-hdfs/ in my case). The heap dump can be opened and investigated with almost any profiler, including NetBeans. Note, however, that the "-XX" ( http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html ) are HostSpot options, they may not work on other JVM implementations. But, afaik, most of the testing is done on Orecle's JVMs 1.6.0_XX, so the option will work and will be helpful in case of OOME problems. As experience shows, if there are no OOMEs, this option does not appear to introduce any problems or performance penalties.
        Hide
        Aaron T. Myers added a comment -

        Hi Ivan, can you comment on what testing you've done of this patch? Did you perhaps modify a test to artificially induce an OOME and then observe that the heap dump showed up in the test output? For that matter, where does the heap dump show up? On the console? In some test output file?

        Show
        Aaron T. Myers added a comment - Hi Ivan, can you comment on what testing you've done of this patch? Did you perhaps modify a test to artificially induce an OOME and then observe that the heap dump showed up in the test output? For that matter, where does the heap dump show up? On the console? In some test output file?
        Aaron T. Myers made changes -
        Assignee Ivan A. Veselovsky [ iveselovsky ]
        Hide
        Ivan A. Veselovsky added a comment -

        This is the change in pom.xml maven config, it should not require new or modified tests.

        Show
        Ivan A. Veselovsky added a comment - This is the change in pom.xml maven config, it should not require new or modified tests.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12546707/HADOOP-8851-vs-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1527//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1527//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546707/HADOOP-8851-vs-trunk.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1527//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1527//console This message is automatically generated.
        Robert Joseph Evans made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Robert Joseph Evans added a comment -

        Submitting so precommit build runs

        Show
        Robert Joseph Evans added a comment - Submitting so precommit build runs
        Ivan A. Veselovsky made changes -
        Field Original Value New Value
        Attachment HADOOP-8851-vs-trunk.patch [ 12546707 ]
        Ivan A. Veselovsky created issue -

          People

          • Assignee:
            Ivan A. Veselovsky
            Reporter:
            Ivan A. Veselovsky
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development