Derby
  1. Derby
  2. DERBY-4571

Memory leak on server when using "SET ROLE" command

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.5.1.1, 10.5.3.0
    • Fix Version/s: 10.5.3.1, 10.6.1.0
    • Component/s: Network Server
    • Labels:
    • Environment:
      Windows XP, Java 6 (build doesn't matter: 10-18)
    • Bug behavior facts:
      Crash, Data corruption

      Description

      Scenario:
      -connect into database
      -use command SET ROLE
      -repeat simple selects many times -> memory consumtions ends with database "outOfMemory: heap space" crash.

      Sometimes this crash ends with loosing (of course previously commited) data in different tables - this is in the case using "UPDATE" etc. command.
      Without command SET ROLE everythings works OK.

      1. clear_deps.diff
        0.6 kB
        Knut Anders Hatlen
      2. D4571.java
        0.6 kB
        Knut Anders Hatlen
      3. DbRoleMemoryLeak.java
        0.8 kB
        Roman Musil
      4. derby-4571-1a.diff
        5 kB
        Knut Anders Hatlen
      5. derby-4571-1a.stat
        0.2 kB
        Knut Anders Hatlen
      6. memRole.sql
        0.8 kB
        Knut Anders Hatlen
      7. memRole.sql
        0.8 kB
        Roman Musil
      8. test.diff
        4 kB
        Knut Anders Hatlen

        Activity

        Gavin made changes -
        Workflow jira [ 12500190 ] Default workflow, editable Closed status [ 12802790 ]
        Knut Anders Hatlen made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Knut Anders Hatlen added a comment -

        [bulk update] Close all resolved issues that haven't been updated for more than one year.

        Show
        Knut Anders Hatlen added a comment - [bulk update] Close all resolved issues that haven't been updated for more than one year.
        Knut Anders Hatlen made changes -
        Fix Version/s 10.5.3.1 [ 12314182 ]
        Hide
        Knut Anders Hatlen added a comment -

        Merged fix to the 10.5 branch and committed revision 927433.

        Show
        Knut Anders Hatlen added a comment - Merged fix to the 10.5 branch and committed revision 927433.
        Knut Anders Hatlen made changes -
        Status In Progress [ 3 ] Resolved [ 5 ]
        Issue & fix info [Patch Available]
        Fix Version/s 10.6.0.0 [ 12313727 ]
        Resolution Fixed [ 1 ]
        Hide
        Knut Anders Hatlen added a comment -

        Thanks for reviewing the patch, Dag.
        Committed revision 918359.

        Show
        Knut Anders Hatlen added a comment - Thanks for reviewing the patch, Dag. Committed revision 918359.
        Hide
        Dag H. Wanvik added a comment -

        Patch looks right to me! Thanks for handling this, Knut.
        +1

        Show
        Dag H. Wanvik added a comment - Patch looks right to me! Thanks for handling this, Knut. +1
        Knut Anders Hatlen made changes -
        Issue & fix info [Patch Available]
        Knut Anders Hatlen made changes -
        Attachment derby-4571-1a.diff [ 12437614 ]
        Attachment derby-4571-1a.stat [ 12437615 ]
        Hide
        Knut Anders Hatlen added a comment -

        Attaching a new patch which includes both the fix and the test case. All the regression tests ran cleanly with this patch.

        As to my comment about performance implications above, I had forgotten that prepared statements reuse activations, so this code will not be invoked as frequently as I thought and it should not be an issue.

        Show
        Knut Anders Hatlen added a comment - Attaching a new patch which includes both the fix and the test case. All the regression tests ran cleanly with this patch. As to my comment about performance implications above, I had forgotten that prepared statements reuse activations, so this code will not be invoked as frequently as I thought and it should not be an issue.
        Hide
        Knut Anders Hatlen added a comment -

        I was a little bit worried about the performance impact of clearing the activation's dependencies unconditionally on each execution, since clearDependencies() ends up synchronizing on a Hashtable and on a BasicDependencyManager, both of which are shared between all threads accessing the same database. However, running the micro-benchmark attached to DERBY-3024 in noshare mode on a 32-way processor did not show any negative impact. I cannot think of any other test that would be more likely to be affected by this, so unless someone has an idea for a simple way to check whether the call to clearDependencies() is needed, I plan to go ahead with the unconditional call to clearDependencies() from BaseActivation.close().

        Show
        Knut Anders Hatlen added a comment - I was a little bit worried about the performance impact of clearing the activation's dependencies unconditionally on each execution, since clearDependencies() ends up synchronizing on a Hashtable and on a BasicDependencyManager, both of which are shared between all threads accessing the same database. However, running the micro-benchmark attached to DERBY-3024 in noshare mode on a 32-way processor did not show any negative impact. I cannot think of any other test that would be more likely to be affected by this, so unless someone has an idea for a simple way to check whether the call to clearDependencies() is needed, I plan to go ahead with the unconditional call to clearDependencies() from BaseActivation.close().
        Knut Anders Hatlen made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Knut Anders Hatlen made changes -
        Attachment test.diff [ 12437486 ]
        Hide
        Knut Anders Hatlen added a comment -

        Attaching patch test.diff which contains a regression test case for this bug. When it's run as part of the junit-lowmem suite, it fails with OOME without the fix and passes with the fix.

        Show
        Knut Anders Hatlen added a comment - Attaching patch test.diff which contains a regression test case for this bug. When it's run as part of the junit-lowmem suite, it fails with OOME without the fix and passes with the fix.
        Knut Anders Hatlen made changes -
        Assignee Knut Anders Hatlen [ knutanders ]
        Knut Anders Hatlen made changes -
        Attachment clear_deps.diff [ 12437349 ]
        Hide
        Knut Anders Hatlen added a comment -

        It looks like the problem goes away if we make BaseActivation.close() clear its dependencies, see the attached patch clear_deps.diff. I haven't run any other tests on the patch yet, but clearing the dependencies on close does sound like the correct thing to do.

        Show
        Knut Anders Hatlen added a comment - It looks like the problem goes away if we make BaseActivation.close() clear its dependencies, see the attached patch clear_deps.diff. I haven't run any other tests on the patch yet, but clearing the dependencies on close does sound like the correct thing to do.
        Hide
        Knut Anders Hatlen added a comment -

        Thanks, I'm able to reproduce the OOME now.

        I'm wondering, could this be because we have made activations dependent on the current role and the dependency table keeps references to activations even after they're no longer in use?

        Show
        Knut Anders Hatlen added a comment - Thanks, I'm able to reproduce the OOME now. I'm wondering, could this be because we have made activations dependent on the current role and the dependency table keeps references to activations even after they're no longer in use?
        Knut Anders Hatlen made changes -
        Attachment memRole.sql [ 12437348 ]
        Hide
        Knut Anders Hatlen added a comment -

        Two of the CALL statements lacked some characters at the end of the line. Uploading a fixed version of memRole.sql.

        Show
        Knut Anders Hatlen added a comment - Two of the CALL statements lacked some characters at the end of the line. Uploading a fixed version of memRole.sql.
        Roman Musil made changes -
        Attachment DbRoleMemoryLeak.java [ 12437226 ]
        Attachment memRole.sql [ 12437227 ]
        Hide
        Roman Musil added a comment -

        If you create database with script memRole.sql and run attached Java program than memory is growing and growing. Execute command "GRANT SELECT ON A TO U1" makes Derby server happy, but revoking this rights from user ("REVOKE SELECT ON A FROM U1") makes DB unhapy again.

        Show
        Roman Musil added a comment - If you create database with script memRole.sql and run attached Java program than memory is growing and growing. Execute command "GRANT SELECT ON A TO U1" makes Derby server happy, but revoking this rights from user ("REVOKE SELECT ON A FROM U1") makes DB unhapy again.
        Hide
        Roman Musil added a comment -

        OK, it is little bit more complicated, than in your attached Java program. The SELECT command must use rights defined by the ROLE. If "SET ROLE" is executed, but SELECT is granted directly to the user, then it works OK.
        I'll provide a sample Java program in a few hours.

        (I know, it has to be two different issue: one "memory leak" and second "data loosing", but the "data loosing" is difficult to reproduce.
        Database is installed "as is" - no special configuration -> no write disk cache.)
        There is the stack trace from derby.log (btw: the database rapidly slows down before exception):
        Exception in thread "DRDAConnThread_8" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2882)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
        at java.lang.StringBuffer.append(StringBuffer.java:224)
        at java.io.StringWriter.write(StringWriter.java:95)
        at java.io.PrintWriter.write(PrintWriter.java:412)
        at java.io.PrintWriter.write(PrintWriter.java:429)
        at java.io.PrintWriter.print(PrintWriter.java:559)
        at org.apache.derby.iapi.error.ErrorStringBuilder.appendln(Unknown Source)
        at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(Unknown Source)
        at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedPreparedStatement.execute(Unknown Source)
        at org.apache.derby.impl.drda.DRDAStatement.execute(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

        Show
        Roman Musil added a comment - OK, it is little bit more complicated, than in your attached Java program. The SELECT command must use rights defined by the ROLE. If "SET ROLE" is executed, but SELECT is granted directly to the user, then it works OK. I'll provide a sample Java program in a few hours. (I know, it has to be two different issue: one "memory leak" and second "data loosing", but the "data loosing" is difficult to reproduce. Database is installed "as is" - no special configuration -> no write disk cache.) There is the stack trace from derby.log (btw: the database rapidly slows down before exception): Exception in thread "DRDAConnThread_8" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) at java.lang.StringBuffer.append(StringBuffer.java:224) at java.io.StringWriter.write(StringWriter.java:95) at java.io.PrintWriter.write(PrintWriter.java:412) at java.io.PrintWriter.write(PrintWriter.java:429) at java.io.PrintWriter.print(PrintWriter.java:559) at org.apache.derby.iapi.error.ErrorStringBuilder.appendln(Unknown Source) at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.execute(Unknown Source) at org.apache.derby.impl.drda.DRDAStatement.execute(Unknown Source) at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source) at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
        Knut Anders Hatlen made changes -
        Field Original Value New Value
        Attachment D4571.java [ 12437065 ]
        Hide
        Knut Anders Hatlen added a comment -

        Hi Roman,

        I tried to reproduce the OOME by following the steps you described, but it won't reproduce in my environment with Derby 10.5.3.0. I ran the attached program with limited heap size (java -Xmx6M). What do I need to change in the program in order to get the OOME?

        Show
        Knut Anders Hatlen added a comment - Hi Roman, I tried to reproduce the OOME by following the steps you described, but it won't reproduce in my environment with Derby 10.5.3.0. I ran the attached program with limited heap size (java -Xmx6M). What do I need to change in the program in order to get the OOME?
        Hide
        Kristian Waagan added a comment -

        Hi Roman,

        Do you happen to have a stack trace from the OOME?
        I know stack traces of OOMEs may be a red herring (due to concurrency / multithreading), but if you're consistently seeing OOMEs in the same spot of of the code it may be a good start for debugging.

        Also, are you sure you're loosing previously committed data?
        If so, do you know if you're running with the disk cache enabled or not?

        Thanks,

        Show
        Kristian Waagan added a comment - Hi Roman, Do you happen to have a stack trace from the OOME? I know stack traces of OOMEs may be a red herring (due to concurrency / multithreading), but if you're consistently seeing OOMEs in the same spot of of the code it may be a good start for debugging. Also, are you sure you're loosing previously committed data? If so, do you know if you're running with the disk cache enabled or not? Thanks,
        Roman Musil created issue -

          People

          • Assignee:
            Knut Anders Hatlen
            Reporter:
            Roman Musil
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development