Derby
  1. Derby
  2. DERBY-5336

Repeated database creation causes OutOfMemoryError

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.8.1.2
    • Fix Version/s: 10.8.2.2, 10.9.1.0
    • Component/s: Services
    • Labels:
      None
    • Environment:
      Windows 7 64-bit, jdk1.6.0_26
    • Urgency:
      Normal
    • Issue & fix info:
      Repro attached, Workaround attached
    • Bug behavior facts:
      Crash, Regression

      Description

      Repeatedly creating, querying, updating, and shutting down a database or databases eventually causes OutOfMemoryError.

      Java source reproducing the problem is attached.

      1. derby-5336-1a-remove_context_on_stop.diff
        0.8 kB
        Kristian Waagan
      2. derby-5336-2a-regression_test.diff
        5 kB
        Kristian Waagan
      3. DerbyOutOfMemoryErrorRepro.java
        3 kB
        Aja Walker
      4. LicensedRepro.java
        3 kB
        Aja Walker

        Issue Links

          Activity

          Aja Walker created issue -
          Hide
          Aja Walker added a comment - - edited

          Attached source code for reproducing the error.

          Show
          Aja Walker added a comment - - edited Attached source code for reproducing the error.
          Aja Walker made changes -
          Field Original Value New Value
          Attachment DerbyOutOfMemoryErrorRepro.java [ 12486511 ]
          Aja Walker made changes -
          Summary Creation of multiple databases causes OutOfMemoryError Repeated database creation causes OutOfMemoryError
          Description Creating, querying, updating, and shutting down multiple databases eventually causes OutOfMemoryError.

          Java source reproducing the problem is attached.
          Repeatedly creating, querying, updating, and shutting down a database or databases eventually causes OutOfMemoryError.

          Java source reproducing the problem is attached.
          Hide
          Knut Anders Hatlen added a comment -

          I'm able to reproduce the OOME on trunk. It looks like the objects belonging to the stopped database instances are still referenced from the index statistics update daemon and cannot be freed.

          Disabling automatic update of index statistics seems to work around the problem. If I run the repro with -Dderby.storage.indexStats.auto=false, I don't see the OOME.

          I'm marking this bug as a regression, since the OOME is not seen with 10.7.1.1.

          Show
          Knut Anders Hatlen added a comment - I'm able to reproduce the OOME on trunk. It looks like the objects belonging to the stopped database instances are still referenced from the index statistics update daemon and cannot be freed. Disabling automatic update of index statistics seems to work around the problem. If I run the repro with -Dderby.storage.indexStats.auto=false, I don't see the OOME. I'm marking this bug as a regression, since the OOME is not seen with 10.7.1.1.
          Knut Anders Hatlen made changes -
          Bug behavior facts [Crash] [Crash, Regression]
          Issue & fix info [Repro attached] [Repro attached, Workaround attached]
          Component/s Services [ 11415 ]
          Component/s Miscellaneous [ 11400 ]
          Hide
          Kristian Waagan added a comment -

          Seems the context used by the istat deamon isn't removed when the database is shut down.
          I'm running regression tests on a possible fix.

          On a related note, I'm wondering if the context class loader should be set to null for the istat daemon thread?

          Show
          Kristian Waagan added a comment - Seems the context used by the istat deamon isn't removed when the database is shut down. I'm running regression tests on a possible fix. On a related note, I'm wondering if the context class loader should be set to null for the istat daemon thread?
          Kristian Waagan made changes -
          Assignee Kristian Waagan [ kristwaa ]
          Kristian Waagan made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          Kristian Waagan added a comment -

          Aja,

          I see that you didn't tick the grant license radio button when uploading the repro.
          Do you mind doing that, such that I can copy most of your repro code to add a regression test for this bug?

          Of course, you can also modify the repro to fit the Derby test framework yourself if you want to (the test should go into the low memory suite)

          Show
          Kristian Waagan added a comment - Aja, I see that you didn't tick the grant license radio button when uploading the repro. Do you mind doing that, such that I can copy most of your repro code to add a regression test for this bug? Of course, you can also modify the repro to fit the Derby test framework yourself if you want to (the test should go into the low memory suite)
          Hide
          Kristian Waagan added a comment -

          Attached patch 1a, which removes the istat daemon context by invoking the cleanup code in the system context.

          Tests passed on Linux and Solaris 11 using Java 6.
          Patch ready for review.

          Show
          Kristian Waagan added a comment - Attached patch 1a, which removes the istat daemon context by invoking the cleanup code in the system context. Tests passed on Linux and Solaris 11 using Java 6. Patch ready for review.
          Kristian Waagan made changes -
          Kristian Waagan made changes -
          Issue & fix info [Workaround attached, Repro attached] [Patch Available, Repro attached, Workaround attached]
          Hide
          Aja Walker added a comment -

          Attaching the repro source with the license box ticked this time.

          Show
          Aja Walker added a comment - Attaching the repro source with the license box ticked this time.
          Aja Walker made changes -
          Attachment LicensedRepro.java [ 12489873 ]
          Hide
          Kristian Waagan added a comment -

          Thanks, Aja.

          Attaching patch 2a, where I copied Aja's repro and modified it to fit with the Derby test framework (JUnit based):

          • switched from DriverManager to DataSource
          • replaced commons io call to delete directory
          • added an extra sanity check
          • added test as testRepeatedDatabaseCreationWithAutoStats in memory.MemoryLeakFixesTest. Should be run with 'ant junit-lowmem' (after building Derby), which uses a 16 MB heap.

          The test failed when run without the fix, it succeeded with patch 1a applied.
          Patch ready for review.

          Show
          Kristian Waagan added a comment - Thanks, Aja. Attaching patch 2a, where I copied Aja's repro and modified it to fit with the Derby test framework (JUnit based): switched from DriverManager to DataSource replaced commons io call to delete directory added an extra sanity check added test as testRepeatedDatabaseCreationWithAutoStats in memory.MemoryLeakFixesTest. Should be run with 'ant junit-lowmem' (after building Derby), which uses a 16 MB heap. The test failed when run without the fix, it succeeded with patch 1a applied. Patch ready for review.
          Kristian Waagan made changes -
          Attachment derby-5336-2a-regression_test.diff [ 12489958 ]
          Hide
          Kristian Waagan added a comment -

          Committed patches 1a and 2a to trunk with revision 1160593.
          Backported to 10.8 with revision 1160595.

          I don't expect more work on this issue.

          Show
          Kristian Waagan added a comment - Committed patches 1a and 2a to trunk with revision 1160593. Backported to 10.8 with revision 1160595. I don't expect more work on this issue.
          Kristian Waagan made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Issue & fix info [Workaround attached, Repro attached, Patch Available] [Repro attached, Workaround attached]
          Fix Version/s 10.8.1.6 [ 12316676 ]
          Fix Version/s 10.9.0.0 [ 12316344 ]
          Resolution Fixed [ 1 ]
          Myrna van Lunteren made changes -
          Fix Version/s 10.8.2.0 [ 12317955 ]
          Fix Version/s 10.8.1.6 [ 12316676 ]
          Knut Anders Hatlen made changes -
          Link This issue is related to DERBY-5412 [ DERBY-5412 ]
          Hide
          Kathey Marsden added a comment -

          The test testRepeadedDatabaseCreationWithAutoStats has a lot of verbosity with -Dderby.test.trace=true

          testRepeatedDatabaseCreationWithAutoStats – 0
          – 1
          – 2
          – 3
          – 4
          – 5
          – 6
          – 7
          – 8
          – 9
          – 10
          – 11
          – 12
          – 13

          Should this traceit call be prntln so it just prnts with -Dderby.tests.debug=true ?
          while (iter < 50) {

          traceit("-- " + iter++);

          Show
          Kathey Marsden added a comment - The test testRepeadedDatabaseCreationWithAutoStats has a lot of verbosity with -Dderby.test.trace=true testRepeatedDatabaseCreationWithAutoStats – 0 – 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 – 11 – 12 – 13 Should this traceit call be prntln so it just prnts with -Dderby.tests.debug=true ? while (iter < 50) { traceit("-- " + iter++);
          Hide
          Kristian Waagan added a comment -

          Yes. I did so with revision 1173647 (trunk only so far).

          I think it may be wise to also reduce the number of iterations, as the test takes a while to complete.

          Show
          Kristian Waagan added a comment - Yes. I did so with revision 1173647 (trunk only so far). I think it may be wise to also reduce the number of iterations, as the test takes a while to complete.
          Hide
          Myrna van Lunteren added a comment -

          Did you plan to backport this latest fix to the 10.8 branch also? If not I'll do it...

          Show
          Myrna van Lunteren added a comment - Did you plan to backport this latest fix to the 10.8 branch also? If not I'll do it...
          Hide
          Kristian Waagan added a comment -

          Thanks for noticing, Myrna.
          Backported to 10.8 with revision 1174576.

          Show
          Kristian Waagan added a comment - Thanks for noticing, Myrna. Backported to 10.8 with revision 1174576.
          Kristian Waagan made changes -
          Link This issue relates to DERBY-5447 [ DERBY-5447 ]
          Knut Anders Hatlen made changes -
          Link This issue is related to DERBY-5472 [ DERBY-5472 ]
          Myrna van Lunteren made changes -
          Fix Version/s 10.8.2.2 [ 12317968 ]
          Fix Version/s 10.8.2.0 [ 12317955 ]
          Hide
          Knut Anders Hatlen added a comment -

          [bulk update] Close all resolved issues that haven't been updated for more than one year.

          Show
          Knut Anders Hatlen added a comment - [bulk update] Close all resolved issues that haven't been updated for more than one year.
          Knut Anders Hatlen made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Workflow jira [ 12620442 ] Default workflow, editable Closed status [ 12799105 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open In Progress In Progress
          25d 14h 24m 1 Kristian Waagan 09/Aug/11 11:47
          In Progress In Progress Resolved Resolved
          13d 21h 36m 1 Kristian Waagan 23/Aug/11 09:23
          Resolved Resolved Closed Closed
          663d 23h 55m 1 Knut Anders Hatlen 17/Jun/13 09:19

            People

            • Assignee:
              Kristian Waagan
              Reporter:
              Aja Walker
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development