Derby
  1. Derby
  2. DERBY-5277

Intermittent OutOfMemoryErrors in BasicSetup.testTriggersWithLOBcolumns()

    Details

      Description

      Seen many times in the JDK 7 tests lately, and also in the Tinderbox. First occurrence was here: http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.7/testing/testlog/sol32/1134678-suitesAll_diff.txt

      (There had been no commits in the last two days before this test run, so it's difficult to say if a recent change caused it.)

      The test case has a comment that says that it should never read the LOB into memory, but according to the stack trace, that's exactly what's happening:

      Caused by: java.lang.OutOfMemoryError: Java heap space
      at org.apache.derby.iapi.types.SQLBinary.readFromStream(Unknown Source)
      at org.apache.derby.iapi.types.SQLBinary.readExternal(Unknown Source)
      at org.apache.derby.iapi.types.SQLBinary.getValue(Unknown Source)
      at org.apache.derby.iapi.types.SQLBinary.loadStream(Unknown Source)
      at org.apache.derby.impl.sql.execute.UpdateResultSet.objectifyStream(Unknown Source)
      at org.apache.derby.impl.sql.execute.UpdateResultSet.collectAffectedRows(Unknown Source)
      at org.apache.derby.impl.sql.execute.UpdateResultSet.open(Unknown Source)
      at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
      at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source)
      at org.apache.derbyTesting.functionTests.tests.upgradeTests.BasicSetup.testTriggersWithLOBcolumns(BasicSetup.java:854)

        Issue Links

          Activity

          Hide
          Knut Anders Hatlen added a comment -

          I added some tracing and it seems that the BLOB is materialized in databases upgraded from any version except 10.7.1.1. So if you upgrade a database from 10.8.1.2 to 10.9, the trigger will still materialize the BLOB.

          Show
          Knut Anders Hatlen added a comment - I added some tracing and it seems that the BLOB is materialized in databases upgraded from any version except 10.7.1.1. So if you upgrade a database from 10.8.1.2 to 10.9, the trigger will still materialize the BLOB.
          Hide
          Mamta A. Satoor added a comment -

          Hi Knut, I have been looking at this jira for sometime and found out the reason behind the problem.

          First of all, you are correct that if the upgrade is from 10.7.1.1 to trunk, we do not run into OOM with the LOB columns if those LOB columns are not needed by the triggering sql and firing triggers. This is because 10.7.1.1 collects information about column in trigger action and hence trunk can be smart about what columns should be read. As for all the other releases, none of them collect information about columns in trigger action and hence trigger firing sql decides to read all the columns including the LOBs even if they are not needed.

          At the time of upgrade, the trigger SPSes get marked invalid correctly.But they do not get recompiled until the time of actual trigger firing. UPDATE sql decides what columns it should read based on what it needs and what the firing triggers will need. When UPDATE sql is collecting all this informaiton about firing triggers, they are not get recompiled and hence UPDATE sql used incorrect information from the invalid triggers.I will work on this jira further to see how it can be resolved. One option would be to have UPDATE sql recognize the fact that the triggers are invalid and hence they should be recompiled first before UPDATE looks at them to determine what columns are needed by the triggers.

          As you noticed in the jira, this problem is intermittent because it all depends on how much memory upgrade suite has to run the tests. I guess I must have had enough memory to not run into OOM when I ran the upgrade suite few times on my machine. One of the things I had wondered about while adding the test in BasicSetup.testTriggersWithLOBcolumns if we could run upgrade suite with lower memory(something like lowmem suite). Better yet would be to just run testTriggersWithLOBcolumns test with lowmem. I will appreciate if anyone knows of a way of doing that in junit framework.

          Show
          Mamta A. Satoor added a comment - Hi Knut, I have been looking at this jira for sometime and found out the reason behind the problem. First of all, you are correct that if the upgrade is from 10.7.1.1 to trunk, we do not run into OOM with the LOB columns if those LOB columns are not needed by the triggering sql and firing triggers. This is because 10.7.1.1 collects information about column in trigger action and hence trunk can be smart about what columns should be read. As for all the other releases, none of them collect information about columns in trigger action and hence trigger firing sql decides to read all the columns including the LOBs even if they are not needed. At the time of upgrade, the trigger SPSes get marked invalid correctly.But they do not get recompiled until the time of actual trigger firing. UPDATE sql decides what columns it should read based on what it needs and what the firing triggers will need. When UPDATE sql is collecting all this informaiton about firing triggers, they are not get recompiled and hence UPDATE sql used incorrect information from the invalid triggers.I will work on this jira further to see how it can be resolved. One option would be to have UPDATE sql recognize the fact that the triggers are invalid and hence they should be recompiled first before UPDATE looks at them to determine what columns are needed by the triggers. As you noticed in the jira, this problem is intermittent because it all depends on how much memory upgrade suite has to run the tests. I guess I must have had enough memory to not run into OOM when I ran the upgrade suite few times on my machine. One of the things I had wondered about while adding the test in BasicSetup.testTriggersWithLOBcolumns if we could run upgrade suite with lower memory(something like lowmem suite). Better yet would be to just run testTriggersWithLOBcolumns test with lowmem. I will appreciate if anyone knows of a way of doing that in junit framework.
          Hide
          Mamta A. Satoor added a comment -

          Trigger action SPSes get marked invalid at the time of upgrade. When such a trigger fires again, we regenerate the trigger action SPS. This regeneration looks at SYSTRIGGERS to find out what columns are being used through the REFERENCING CLAUSE and uses them to find the relative column positions of those columns in the regenerated trigger action. At the time of upgrade, this information will be available in SYSTRIGGER if the trigger was created in 10.7.1.1. But for all the other releases, SYSTRIGGERS does not have that information because that informaiton was never collected for the triggers in those releases. Because of this, the regenerated trigger action SPS will assume that all the columns will be read from the trigger table and hence it will use the absolute column positions of those columns in the trigger table in the regenerated trigger action SPS. This will cause such triggers to not be able to use the performance enhancement of selective column reading from the trigger table and hence even though not all the columns from the trigger table table are required by the triggering sql and firing triggers, we will read all the columns. One fix for this can be that when a trigger fires and it finds that it's trigger action SPS is invalid, then trigger should check if it uses REFERENCING clause and if the trigger action column information is missing from the SYSTRIGGERS, then it should drop and recreate the trigger with the trigger action column information collected and then regenerate the trigger action SPS based on it. Doing this will make Derby not read the columns not needed by the firing trigger and triggering sql. I will look into this to see what work is involved in collecting the trigger action column information for the triggers with that info missing. This should only happen for the triggers which are not created in 10.9 and higher because only releases prior to 10.9 (with the exception of 10.7.1.1) didn't collect the information about trigger action columns.

          Show
          Mamta A. Satoor added a comment - Trigger action SPSes get marked invalid at the time of upgrade. When such a trigger fires again, we regenerate the trigger action SPS. This regeneration looks at SYSTRIGGERS to find out what columns are being used through the REFERENCING CLAUSE and uses them to find the relative column positions of those columns in the regenerated trigger action. At the time of upgrade, this information will be available in SYSTRIGGER if the trigger was created in 10.7.1.1. But for all the other releases, SYSTRIGGERS does not have that information because that informaiton was never collected for the triggers in those releases. Because of this, the regenerated trigger action SPS will assume that all the columns will be read from the trigger table and hence it will use the absolute column positions of those columns in the trigger table in the regenerated trigger action SPS. This will cause such triggers to not be able to use the performance enhancement of selective column reading from the trigger table and hence even though not all the columns from the trigger table table are required by the triggering sql and firing triggers, we will read all the columns. One fix for this can be that when a trigger fires and it finds that it's trigger action SPS is invalid, then trigger should check if it uses REFERENCING clause and if the trigger action column information is missing from the SYSTRIGGERS, then it should drop and recreate the trigger with the trigger action column information collected and then regenerate the trigger action SPS based on it. Doing this will make Derby not read the columns not needed by the firing trigger and triggering sql. I will look into this to see what work is involved in collecting the trigger action column information for the triggers with that info missing. This should only happen for the triggers which are not created in 10.9 and higher because only releases prior to 10.9 (with the exception of 10.7.1.1) didn't collect the information about trigger action columns.
          Hide
          Mamta A. Satoor added a comment -

          Until DERBY-5294 (Triggers created prior to 10.9 release will continue to read all the columns from trigger table even after database has been upgraded to 10.9 and higher) is fixed, we risk running into OOM problems BasicSetup.testTriggersWithLOBcolumns(). Once DERBY-5294 is fixed, we can enable BasicSetup.testTriggersWithLOBcolumns() again.

          Show
          Mamta A. Satoor added a comment - Until DERBY-5294 (Triggers created prior to 10.9 release will continue to read all the columns from trigger table even after database has been upgraded to 10.9 and higher) is fixed, we risk running into OOM problems BasicSetup.testTriggersWithLOBcolumns(). Once DERBY-5294 is fixed, we can enable BasicSetup.testTriggersWithLOBcolumns() again.
          Hide
          Mamta A. Satoor added a comment -

          I checked Sun's tinderbox runs and the last 3 runs have run fine with no OOM

          Show
          Mamta A. Satoor added a comment - I checked Sun's tinderbox runs and the last 3 runs have run fine with no OOM
          Hide
          Kathey Marsden added a comment -

          reopen to add backport reject label as this only affects trunk

          Show
          Kathey Marsden added a comment - reopen to add backport reject label as this only affects trunk

            People

            • Assignee:
              Mamta A. Satoor
              Reporter:
              Knut Anders Hatlen
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development