Derby
  1. Derby
  2. DERBY-4200

client side OutOfMemoryError running derbnetclientmats:jdbcapi/derbyStress

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.5.2.0
    • Component/s: Network Client, Test
    • Labels:
      None
    • Environment:
    • Urgency:
      Normal
    • Bug behavior facts:
      Crash

      Description

      On the nightly run for 4/27 - 10.5.1.2 - (769232), I saw client jdbcapi/derbystress.java run out of heap space. The test has not failed like this before on the same machine with the same JVM, and the one checkin on that day DERBY-3991 could not account for this failure.

      I will attach the javacore and heapdump. Taking a quick look at the heap dump, it seems to have a lot of client side Statement objects, which seems to be just the leak the test is checking for. Note: the test runs with 64MB heap. It would be interesting to run with other jvms and force a gc() and a heap dump at this point in the test and see if we still have a lot of Statement objects or if this is a specific platform/JVM issue.

      The trace at the time of failure was :
      1XMCURTHDINFO Current Thread Details
      NULL ----------------------
      3XMTHREADINFO "main" (TID:0x0808D300, sys_thread_t:0x0805CBC8, state:R, native ID:0x0000644F) prio=5
      4XESTACKTRACE at org/apache/derby/client/am/Cursor.allocateCharBuffer(Bytecode PC:77(Compiled Code))
      4XESTACKTRACE at org/apache/derby/client/net/NetStatementReply.parseSQLDTARDarray(Bytecode PC:77(Compiled Code))
      4XESTACKTRACE at org/apache/derby/client/net/NetStatementReply.parseQRYDSC(Bytecode PC:10(Compiled Code))
      4XESTACKTRACE at org/apache/derby/client/net/NetStatementReply.parseOpenQuery(Bytecode PC:104(Compiled Code))
      4XESTACKTRACE at org/apache/derby/client/net/NetStatementReply.parseOPNQRYreply(Bytecode PC:14(Compiled Code))
      4XESTACKTRACE at org/apache/derby/client/net/NetStatementReply.readOpenQuery(Bytecode PC:6(Compiled Code))
      4XESTACKTRACE at org/apache/derby/client/net/StatementReply.readOpenQuery(Bytecode PC:7(Compiled Code))
      4XESTACKTRACE at org/apache/derby/client/net/NetStatement.readOpenQuery_(Bytecode PC:11(Compiled Code))
      4XESTACKTRACE at org/apache/derby/client/am/Statement.readOpenQuery(Bytecode PC:6(Compiled Code))
      4XESTACKTRACE at org/apache/derby/client/am/Statement.flowExecute(Bytecode PC:581(Compiled Code))
      4XESTACKTRACE at org/apache/derby/client/am/Statement.executeQueryX(Bytecode PC:3(Compiled Code))
      4XESTACKTRACE at org/apache/derby/client/am/Statement.executeQuery(Bytecode PC:3(Compiled Code))
      4XESTACKTRACE at org/apache/derbyTesting/functionTests/tests/jdbcapi/derbyStress.testDerby3316(derbyStress.java:156)
      4XESTACKTRACE at org/apache/derbyTesting/functionTests/tests/jdbcapi/derbyStress.main(derbyStress.java:57(Compiled Code))

      1. vmware10_5_SR10HeapDump.phd
        533 kB
        Mamta A. Satoor
      2. vmware10_5_SR7HeapDump.phd
        1.30 MB
        Mamta A. Satoor
      3. windows10_5_HeapDump.phd
        334 kB
        Mamta A. Satoor
      4. heapdump.20090428.084024.25679.phd
        2.29 MB
        Kathey Marsden
      5. javacore.20090428.084024.25679.txt
        168 kB
        Kathey Marsden

        Issue Links

          Activity

          Hide
          Kathey Marsden added a comment -

          javacore and heapdump at the time of failure. To analyze the heapdump you can use the heap analyzer available at: http://www.alphaworks.ibm.com/tech/heapanalyzer

          Show
          Kathey Marsden added a comment - javacore and heapdump at the time of failure. To analyze the heapdump you can use the heap analyzer available at: http://www.alphaworks.ibm.com/tech/heapanalyzer
          Hide
          Kathey Marsden added a comment -

          If you would like to produce a heap dump on another machine at the same point in the test with the IBM 1.6 jvm you can gc() and then call
          com.ibm.jvm.Dump.HeapDump();
          com.ibm.jvm.Dump.JavaDump();

          The same tool mentioned in the previous comment can read the heap dump.

          Then you can see if those Statement objects are present on other platforms. Perhaps you could also just run with a slightly smaller heap set in derbyStress_app.properties to get the OOM.

          Show
          Kathey Marsden added a comment - If you would like to produce a heap dump on another machine at the same point in the test with the IBM 1.6 jvm you can gc() and then call com.ibm.jvm.Dump.HeapDump(); com.ibm.jvm.Dump.JavaDump(); The same tool mentioned in the previous comment can read the heap dump. Then you can see if those Statement objects are present on other platforms. Perhaps you could also just run with a slightly smaller heap set in derbyStress_app.properties to get the OOM.
          Hide
          Rick Hillegas added a comment -

          triaged July 2, 2009

          Show
          Rick Hillegas added a comment - triaged July 2, 2009
          Hide
          Mamta A. Satoor added a comment -

          I changed derbyStress.java's testDerby3316() method to gc() and then to dump the heap in my 10.5 codeline. I ran this changed derbyStress in following 3 scenarios
          1)my Windows XP box with IBM 1.5 jdk. Heap dump is attached as windows10_5_HeapDump.phd
          2)on the vmware machine(vmware is where we have been seeing intermittent out of memory errors) using IBM 1.5 SR7 jdk. Heap dump is attached as vmware10_5_SR7HeapDump.phd
          3)on the vmware using IBM 1.5 SR10 jdk. Heap dump is attached as vmware10_5_SR10HeapDump.phd

          In each of these dumps, using the HeapAnalyzer, I searched for org/apache/derby/client/net/NetStatement objects.
          1)On Windows XP box, I only found 9 org/apache/derby/client/net/NetStatement objects
          2)On vmware with IBM 1.5 SR7, I found 310 org/apache/derby/client/net/NetStatement objects
          3)On vmware with IBM 1.5 SR10, I found 80 org/apache/derby/client/net/NetStatement objects

          So, there is definitely improvement over number of objects from IBM 1.5 SR7 to IBM 1.5 SR10 on vmware. I wonder if the experiment above makes this more of a platform issue vs regression?

          BTW, after my changes, the derbyStress method in question looks as follows(the changes are 3 lines under the comment //Following 3 lines are to dump the heap)
          public static void testDerby3316() throws Exception {
          System.out.println("DERBY-3316: Multiple statement executions ");
          Connection conn = ij.startJBMS();

          Statement s = conn.createStatement();
          s.executeUpdate("CREATE TABLE TAB (col1 varchar(32672))");
          PreparedStatement ps = conn.prepareStatement("INSERT INTO TAB VALUES");
          ps.setString(1,"hello");
          ps.executeUpdate();
          ps.setString(1,"hello");
          ps.executeUpdate();
          ps.close();
          for (int i = 0; i < 2000; i++)

          { s = conn.createStatement(); ResultSet rs = s.executeQuery("SELECT * from tab"); // drain the resultset while (rs.next()); // With DERBY-3316, If I don't explicitly close the resultset or // statement, we get a leak. //rs.close(); //s.close(); }

          //Following 3 lines are to dump the heap
          System.gc();
          com.ibm.jvm.Dump.HeapDump();
          com.ibm.jvm.Dump.JavaDump();
          // close the connection to free up all the result sets that our sloppy
          // user didn't close.
          conn.close();
          conn = ij.startJBMS();
          s = conn.createStatement();
          s.executeUpdate("DROP TABLE TAB");
          s.close();
          conn.close();
          }

          Show
          Mamta A. Satoor added a comment - I changed derbyStress.java's testDerby3316() method to gc() and then to dump the heap in my 10.5 codeline. I ran this changed derbyStress in following 3 scenarios 1)my Windows XP box with IBM 1.5 jdk. Heap dump is attached as windows10_5_HeapDump.phd 2)on the vmware machine(vmware is where we have been seeing intermittent out of memory errors) using IBM 1.5 SR7 jdk. Heap dump is attached as vmware10_5_SR7HeapDump.phd 3)on the vmware using IBM 1.5 SR10 jdk. Heap dump is attached as vmware10_5_SR10HeapDump.phd In each of these dumps, using the HeapAnalyzer, I searched for org/apache/derby/client/net/NetStatement objects. 1)On Windows XP box, I only found 9 org/apache/derby/client/net/NetStatement objects 2)On vmware with IBM 1.5 SR7, I found 310 org/apache/derby/client/net/NetStatement objects 3)On vmware with IBM 1.5 SR10, I found 80 org/apache/derby/client/net/NetStatement objects So, there is definitely improvement over number of objects from IBM 1.5 SR7 to IBM 1.5 SR10 on vmware. I wonder if the experiment above makes this more of a platform issue vs regression? BTW, after my changes, the derbyStress method in question looks as follows(the changes are 3 lines under the comment //Following 3 lines are to dump the heap) public static void testDerby3316() throws Exception { System.out.println(" DERBY-3316 : Multiple statement executions "); Connection conn = ij.startJBMS(); Statement s = conn.createStatement(); s.executeUpdate("CREATE TABLE TAB (col1 varchar(32672))"); PreparedStatement ps = conn.prepareStatement("INSERT INTO TAB VALUES "); ps.setString(1,"hello"); ps.executeUpdate(); ps.setString(1,"hello"); ps.executeUpdate(); ps.close(); for (int i = 0; i < 2000; i++) { s = conn.createStatement(); ResultSet rs = s.executeQuery("SELECT * from tab"); // drain the resultset while (rs.next()); // With DERBY-3316, If I don't explicitly close the resultset or // statement, we get a leak. //rs.close(); //s.close(); } //Following 3 lines are to dump the heap System.gc(); com.ibm.jvm.Dump.HeapDump(); com.ibm.jvm.Dump.JavaDump(); // close the connection to free up all the result sets that our sloppy // user didn't close. conn.close(); conn = ij.startJBMS(); s = conn.createStatement(); s.executeUpdate("DROP TABLE TAB"); s.close(); conn.close(); }
          Hide
          Mamta A. Satoor added a comment - - edited

          On Aug 13th comment, I mentioned trying dumps *after* we get out of 2000 iterations of executing the same query repeatedly. Looking back at the OOM failure, we get exception within the loop. Based on that, I changed the testDerby3316() in derbyStress as follows. In the code below, I added the dumps within the loop every 100 iterations rather than one dump outside of the loop.
          public static void testDerby3316() throws Exception {
          System.out.println("DERBY-3316: Multiple statement executions ");
          Connection conn = ij.startJBMS();

          Statement s = conn.createStatement();
          s.executeUpdate("CREATE TABLE TAB (col1 varchar(32672))");
          PreparedStatement ps = conn.prepareStatement("INSERT INTO TAB VALUES");
          ps.setString(1,"hello");
          ps.executeUpdate();
          ps.setString(1,"hello");
          ps.executeUpdate();
          ps.close();
          for (int i = 0; i < 2000; i++)
          {
          s = conn.createStatement();
          ResultSet rs = s.executeQuery("SELECT * from tab");
          if (i%100 == 0)

          { System.out.println("Iteration number " + i); System.gc(); // System.runFinalization(); com.ibm.jvm.Dump.HeapDump(); com.ibm.jvm.Dump.JavaDump(); }

          // drain the resultset
          while (rs.next());
          // With DERBY-3316, If I don't explicitly close the resultset or
          // statement, we get a leak.
          //rs.close();
          //s.close();
          }
          // close the connection to free up all the result sets that our sloppy
          // user didn't close.
          conn.close();
          conn = ij.startJBMS();
          s = conn.createStatement();
          s.executeUpdate("DROP TABLE TAB");
          s.close();
          conn.close();
          }

          With the code above, when I run it on my windows XP machine with ibm15, the number of org/apache/derby/client/net/NetStatement range from 2 to 102. But the results on vmware is that the number of org/apache/derby/client/net/NetStatement keep increasing in the dump every 100 iterations. It starts with about 4 objects and keeps going up by 100 and right about when the number of objects get to 940+ or so, it runs in OOM. The OOM errors are intermittent but whenever I saw them, it was always when it got to about that 900+ number.

          Talked to Mike about this behavior and he suggested that may be I try to put a sleep after the gc because depending on the system/vc, a call to gc may just schedule a background gc and not necessarily mean objects are being claimed when the call to gc returns. I will get that gc/sleep combo a try and see how that goes.

          Show
          Mamta A. Satoor added a comment - - edited On Aug 13th comment, I mentioned trying dumps * after * we get out of 2000 iterations of executing the same query repeatedly. Looking back at the OOM failure, we get exception within the loop. Based on that, I changed the testDerby3316() in derbyStress as follows. In the code below, I added the dumps within the loop every 100 iterations rather than one dump outside of the loop. public static void testDerby3316() throws Exception { System.out.println(" DERBY-3316 : Multiple statement executions "); Connection conn = ij.startJBMS(); Statement s = conn.createStatement(); s.executeUpdate("CREATE TABLE TAB (col1 varchar(32672))"); PreparedStatement ps = conn.prepareStatement("INSERT INTO TAB VALUES "); ps.setString(1,"hello"); ps.executeUpdate(); ps.setString(1,"hello"); ps.executeUpdate(); ps.close(); for (int i = 0; i < 2000; i++) { s = conn.createStatement(); ResultSet rs = s.executeQuery("SELECT * from tab"); if (i%100 == 0) { System.out.println("Iteration number " + i); System.gc(); // System.runFinalization(); com.ibm.jvm.Dump.HeapDump(); com.ibm.jvm.Dump.JavaDump(); } // drain the resultset while (rs.next()); // With DERBY-3316 , If I don't explicitly close the resultset or // statement, we get a leak. //rs.close(); //s.close(); } // close the connection to free up all the result sets that our sloppy // user didn't close. conn.close(); conn = ij.startJBMS(); s = conn.createStatement(); s.executeUpdate("DROP TABLE TAB"); s.close(); conn.close(); } With the code above, when I run it on my windows XP machine with ibm15, the number of org/apache/derby/client/net/NetStatement range from 2 to 102. But the results on vmware is that the number of org/apache/derby/client/net/NetStatement keep increasing in the dump every 100 iterations. It starts with about 4 objects and keeps going up by 100 and right about when the number of objects get to 940+ or so, it runs in OOM. The OOM errors are intermittent but whenever I saw them, it was always when it got to about that 900+ number. Talked to Mike about this behavior and he suggested that may be I try to put a sleep after the gc because depending on the system/vc, a call to gc may just schedule a background gc and not necessarily mean objects are being claimed when the call to gc returns. I will get that gc/sleep combo a try and see how that goes.
          Hide
          Mamta A. Satoor added a comment -

          Wanted to provide jvm versions for testing results above

          The jvm version on Windows XP is as follows
          java version "1.5.0"
          Java(TM) 2 Runtime Environment, Standard Edition (build pwi32dev-20080315 (SR7))

          IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32 j9vmwi3223-20080315 (JIT enabled)
          J9VM - 20080314_17962_lHdSMr
          JIT - 20080130_0718ifx2_r8
          GC - 200802_08)
          JCL - 20080314

          On vmware machine, the jvm version is as follows
          java version "1.5.0"
          Java(TM) 2 Runtime Environment, Standard Edition (build pxi32devifx-20070806 (SR5a))
          IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux x86-32 j9vmxi3223-20070426 (JIT enabled)
          J9VM - 20070420_12448_lHdSMR
          JIT - 20070419_1806_r8
          GC - 200704_19)
          JCL - 20070725

          I will try to find if there is a newer version of 10.5 available on vmware machine

          Show
          Mamta A. Satoor added a comment - Wanted to provide jvm versions for testing results above The jvm version on Windows XP is as follows java version "1.5.0" Java(TM) 2 Runtime Environment, Standard Edition (build pwi32dev-20080315 (SR7)) IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32 j9vmwi3223-20080315 (JIT enabled) J9VM - 20080314_17962_lHdSMr JIT - 20080130_0718ifx2_r8 GC - 200802_08) JCL - 20080314 On vmware machine, the jvm version is as follows java version "1.5.0" Java(TM) 2 Runtime Environment, Standard Edition (build pxi32devifx-20070806 (SR5a)) IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux x86-32 j9vmxi3223-20070426 (JIT enabled) J9VM - 20070420_12448_lHdSMR JIT - 20070419_1806_r8 GC - 200704_19) JCL - 20070725 I will try to find if there is a newer version of 10.5 available on vmware machine
          Hide
          Mamta A. Satoor added a comment -

          I tried the combination of gc and sleep before doing the dump of heap and that definitely helped with bringing down the number of objects on vmware. I also did not run into any OOM exceptions even after running the tests over 20times. To me, it esems like the issue is with garbage collection thread on vmware. I am still looking for a more updated IBM15 on vmware. Will run the tests on that jdk to see if it helps with garbage collection better.

          The new code that I tried looks as follows
          public static void testDerby3316() throws Exception {
          System.out.println("DERBY-3316: Multiple statement executions ");
          Connection conn = ij.startJBMS();

          Statement s = conn.createStatement();
          s.executeUpdate("CREATE TABLE TAB (col1 varchar(32672))");
          PreparedStatement ps = conn.prepareStatement("INSERT INTO TAB VALUES");
          ps.setString(1,"hello");
          ps.executeUpdate();
          ps.setString(1,"hello");
          ps.executeUpdate();
          ps.close();
          for (int i = 0; i < 2000; i++)
          {
          s = conn.createStatement();
          ResultSet rs = s.executeQuery("SELECT * from tab");
          if (i%100 == 0)

          { System.out.println("Iteration number " + i); System.gc(); // System.runFinalization(); Thread.sleep(1000); com.ibm.jvm.Dump.HeapDump(); com.ibm.jvm.Dump.JavaDump(); }

          // drain the resultset
          while (rs.next());
          // With DERBY-3316, If I don't explicitly close the resultset or
          // statement, we get a leak.
          //rs.close();
          //s.close();
          }
          // close the connection to free up all the result sets that our sloppy
          // user didn't close.
          conn.close();
          conn = ij.startJBMS();
          s = conn.createStatement();
          s.executeUpdate("DROP TABLE TAB");
          s.close();
          conn.close();
          }

          Show
          Mamta A. Satoor added a comment - I tried the combination of gc and sleep before doing the dump of heap and that definitely helped with bringing down the number of objects on vmware. I also did not run into any OOM exceptions even after running the tests over 20times. To me, it esems like the issue is with garbage collection thread on vmware. I am still looking for a more updated IBM15 on vmware. Will run the tests on that jdk to see if it helps with garbage collection better. The new code that I tried looks as follows public static void testDerby3316() throws Exception { System.out.println(" DERBY-3316 : Multiple statement executions "); Connection conn = ij.startJBMS(); Statement s = conn.createStatement(); s.executeUpdate("CREATE TABLE TAB (col1 varchar(32672))"); PreparedStatement ps = conn.prepareStatement("INSERT INTO TAB VALUES "); ps.setString(1,"hello"); ps.executeUpdate(); ps.setString(1,"hello"); ps.executeUpdate(); ps.close(); for (int i = 0; i < 2000; i++) { s = conn.createStatement(); ResultSet rs = s.executeQuery("SELECT * from tab"); if (i%100 == 0) { System.out.println("Iteration number " + i); System.gc(); // System.runFinalization(); Thread.sleep(1000); com.ibm.jvm.Dump.HeapDump(); com.ibm.jvm.Dump.JavaDump(); } // drain the resultset while (rs.next()); // With DERBY-3316 , If I don't explicitly close the resultset or // statement, we get a leak. //rs.close(); //s.close(); } // close the connection to free up all the result sets that our sloppy // user didn't close. conn.close(); conn = ij.startJBMS(); s = conn.createStatement(); s.executeUpdate("DROP TABLE TAB"); s.close(); conn.close(); }
          Hide
          Mamta A. Satoor added a comment -

          I was able to get hold of IBM1.5 SR10 on both my windows XP laptop and on the vmware machine and in both places, the behavior stayed the same which is no OOM problems on windows XP but intermittent OOMs on vmware continued. Whenever we ran into OOM on vmware, it would be close to after 900 to 1000 iterations of ResultSet rs = s.executeQuery("SELECT * from tab"); that we would get OOM. And the number of NetStatement objects in the heapdumps consistently is over 900+ when we run into OOM.

          I will wait to hear what community thinks of these experiments. It definitely appears to be specific to this vmware platform which has single processor.

          Show
          Mamta A. Satoor added a comment - I was able to get hold of IBM1.5 SR10 on both my windows XP laptop and on the vmware machine and in both places, the behavior stayed the same which is no OOM problems on windows XP but intermittent OOMs on vmware continued. Whenever we ran into OOM on vmware, it would be close to after 900 to 1000 iterations of ResultSet rs = s.executeQuery("SELECT * from tab"); that we would get OOM. And the number of NetStatement objects in the heapdumps consistently is over 900+ when we run into OOM. I will wait to hear what community thinks of these experiments. It definitely appears to be specific to this vmware platform which has single processor.
          Hide
          Kathey Marsden added a comment -

          Mamta, if you put a sleep and force gc() every 100 iterations, does the problem occur on the vmware machien?

          Show
          Kathey Marsden added a comment - Mamta, if you put a sleep and force gc() every 100 iterations, does the problem occur on the vmware machien?
          Hide
          Mamta A. Satoor added a comment - - edited

          On vmware, I have not run into OOM if there is sleep and gc after every 100 iterations.

          Show
          Mamta A. Satoor added a comment - - edited On vmware, I have not run into OOM if there is sleep and gc after every 100 iterations.
          Hide
          Kathey Marsden added a comment -

          My conclusion then would be that there is not a Derby bug here. I am not sure, however, if the JVM should detect that we are about to run out of memory and make the active thread yield and allow the garbage collector to do its work before continuing.

          Show
          Kathey Marsden added a comment - My conclusion then would be that there is not a Derby bug here. I am not sure, however, if the JVM should detect that we are about to run out of memory and make the active thread yield and allow the garbage collector to do its work before continuing.
          Hide
          Mamta A. Satoor added a comment -

          After the work on this jira entry, it appears that this more than likely is a jvm/platform issue rather than a Derby bug. Closing the jira based on this.

          Show
          Mamta A. Satoor added a comment - After the work on this jira entry, it appears that this more than likely is a jvm/platform issue rather than a Derby bug. Closing the jira based on this.
          Hide
          Kathey Marsden added a comment -

          I was able to reproduce the problem 38 out of 50 runs by restricting the test to a single CPU with:

          taskset 0x00000001 java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest jdbcapi/derbyStress.java

          I will play with sleep values to see what is appropriate to let the finalizer catch up.

          Show
          Kathey Marsden added a comment - I was able to reproduce the problem 38 out of 50 runs by restricting the test to a single CPU with: taskset 0x00000001 java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest jdbcapi/derbyStress.java I will play with sleep values to see what is appropriate to let the finalizer catch up.
          Hide
          Kathey Marsden added a comment -

          Since we have only ever seen this on the IBM jvms, I started to wonder if this might in fact be a jvm bug. I wondered if perhaps the garbage collector should wait until the finalizer catches up before throwing the out of memory. To experiment I tried making a class with a sleep in the finalize method to see if the Sun JVM waits but it does not wait in this case and runs out of memory with the sleep. Here is the code I used:
          public class GCWontWait {

          private static StringBuffer BIG_STRING_BUFFER;

          static

          { BIG_STRING_BUFFER = new StringBuffer(30000); for (int i = 0; i<1000; i++) BIG_STRING_BUFFER.append("123456789012345678901234567890"); }

          public static void main(String[] args) {

          SlowString s;
          for (int i=0;i < 100000;i++)

          { s = new SlowString(BIG_STRING_BUFFER); if ((i % 100) == 0) System.out.println("i = " + i); }

          }

          }

          public class SlowString {

          String v;

          public SlowString(StringBuffer s)

          { v = new String(s); }

          public void finalize() throws Throwable

          { Thread.sleep(10); }

          }

          java -Xmx16m -cp . GCWontWait

          With the Sun JVM failed with OOM after only 200 instantiations
          java version "1.6.0_21"
          Java(TM) SE Runtime Environment (build 1.6.0_21-b07)
          Java HotSpot(TM) Client VM (build 17.0-b17, mixed mode)

          IBM JDK 1.6 actually did much better on the allocation and instantiated all 100000 objects.
          java version "1.6.0"
          Java(TM) SE Runtime Environment (build pwi3260sr9fp1-20110208_03(SR9 FP1))
          IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260sr9-201102
          03_74623 (JIT enabled, AOT enabled)
          J9VM - 20110203_074623
          JIT - r9_20101028_17488ifx3
          GC - 20101027_AA)
          JCL - 20110203_01

          I am not sure what this means except that Sun doesn't wait for the finalizer either. Maybe the IBM jvm is just reusing the duplicate strings more efficiently.

          Show
          Kathey Marsden added a comment - Since we have only ever seen this on the IBM jvms, I started to wonder if this might in fact be a jvm bug. I wondered if perhaps the garbage collector should wait until the finalizer catches up before throwing the out of memory. To experiment I tried making a class with a sleep in the finalize method to see if the Sun JVM waits but it does not wait in this case and runs out of memory with the sleep. Here is the code I used: public class GCWontWait { private static StringBuffer BIG_STRING_BUFFER; static { BIG_STRING_BUFFER = new StringBuffer(30000); for (int i = 0; i<1000; i++) BIG_STRING_BUFFER.append("123456789012345678901234567890"); } public static void main(String[] args) { SlowString s; for (int i=0;i < 100000;i++) { s = new SlowString(BIG_STRING_BUFFER); if ((i % 100) == 0) System.out.println("i = " + i); } } } public class SlowString { String v; public SlowString(StringBuffer s) { v = new String(s); } public void finalize() throws Throwable { Thread.sleep(10); } } java -Xmx16m -cp . GCWontWait With the Sun JVM failed with OOM after only 200 instantiations java version "1.6.0_21" Java(TM) SE Runtime Environment (build 1.6.0_21-b07) Java HotSpot(TM) Client VM (build 17.0-b17, mixed mode) IBM JDK 1.6 actually did much better on the allocation and instantiated all 100000 objects. java version "1.6.0" Java(TM) SE Runtime Environment (build pwi3260sr9fp1-20110208_03(SR9 FP1)) IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260sr9-201102 03_74623 (JIT enabled, AOT enabled) J9VM - 20110203_074623 JIT - r9_20101028_17488ifx3 GC - 20101027_AA) JCL - 20110203_01 I am not sure what this means except that Sun doesn't wait for the finalizer either. Maybe the IBM jvm is just reusing the duplicate strings more efficiently.
          Hide
          Kathey Marsden added a comment -

          Yes if I make each one of the strings slightly different IBM also fails after about 200 instantiations.
          $ java -Xmx16M -cp . GCWontWait
          i = 0
          i = 100
          i = 200
          JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError
          " - please wait.

          Show
          Kathey Marsden added a comment - Yes if I make each one of the strings slightly different IBM also fails after about 200 instantiations. $ java -Xmx16M -cp . GCWontWait i = 0 i = 100 i = 200 JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError " - please wait.
          Hide
          Kathey Marsden added a comment -

          For trunk, this issue is fixed with the conversion of the test to JUnit DERBY-3337.

          Changes from 1141769/1141923 to 1141927:
          ------------------------------------------------------------------------
          r1141924 | kmarsden | 2011-07-01 14:54:30 +0200 (Fri, 01 Jul 2011) | 7 lines

          DERBY-3337 convert jdbcapi/derbyStress.java to JUnit
          DERBY-4200 client side OutOfMemoryError running derbnetclientmats:jdbcapi/derbyStress

          Convert test to JUnit and also run the finalizer periodically if freeMemory() falls below 500K. The converted tests is no longer called derbyStress. It is now memory.MemoryLeakFixes and runs with the junit-lowmem target or can be run directly with junit.textui.TestRunner with -Xmx16M.

          I will leave it open for a bit while I backport to 10.5

          Show
          Kathey Marsden added a comment - For trunk, this issue is fixed with the conversion of the test to JUnit DERBY-3337 . Changes from 1141769/1141923 to 1141927: ------------------------------------------------------------------------ r1141924 | kmarsden | 2011-07-01 14:54:30 +0200 (Fri, 01 Jul 2011) | 7 lines DERBY-3337 convert jdbcapi/derbyStress.java to JUnit DERBY-4200 client side OutOfMemoryError running derbnetclientmats:jdbcapi/derbyStress Convert test to JUnit and also run the finalizer periodically if freeMemory() falls below 500K. The converted tests is no longer called derbyStress. It is now memory.MemoryLeakFixes and runs with the junit-lowmem target or can be run directly with junit.textui.TestRunner with -Xmx16M. I will leave it open for a bit while I backport to 10.5
          Hide
          Kathey Marsden added a comment -

          reopening because ended up being a test issue.

          Show
          Kathey Marsden added a comment - reopening because ended up being a test issue.
          Hide
          Kathey Marsden added a comment -

          The problem in this case was that the finalizer work was not complete when the OOM was thrown. With DERBY-3337 the conversion of derbyStress, the test was changed to call Runtime.runFinalization() if it ran low on memory thus avoiding the OOM.
          The new test has been moved to the memory suite and is called MemoryLeakFixesTest. The memory suite should be run with -Xmx16M or with the ant junit-lowmem target to make sure it runs with a low maximum heap.

          Show
          Kathey Marsden added a comment - The problem in this case was that the finalizer work was not complete when the OOM was thrown. With DERBY-3337 the conversion of derbyStress, the test was changed to call Runtime.runFinalization() if it ran low on memory thus avoiding the OOM. The new test has been moved to the memory suite and is called MemoryLeakFixesTest. The memory suite should be run with -Xmx16M or with the ant junit-lowmem target to make sure it runs with a low maximum heap.
          Hide
          Knut Anders Hatlen added a comment -

          [bulk update] Close all resolved issues that haven't been updated for more than one year.

          Show
          Knut Anders Hatlen added a comment - [bulk update] Close all resolved issues that haven't been updated for more than one year.

            People

            • Assignee:
              Kathey Marsden
              Reporter:
              Kathey Marsden
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development