Derby
  1. Derby
  2. DERBY-759

Query with thousands of logical operators and parameters causes StackOverflowError and then ERROR 40XT0: An internal error was identified by RawStore module on next statement

    Details

    • Urgency:
      Normal
    • Bug behavior facts:
      Crash

      Description

      An SQL Statement with thousands of logical operators and parameters causes a StackOverflow Error and then the next statement throws a ERROR 40XT0: An internal error was identified by RawStore module.

      The statement is of the form SELECT * FROM T0 WHERE (si = ? AND si = ? ) OR (si = ? AND si = ? ) ...

      I came across this issue when looking at DERBY-739 and code generation issues for large queries, thus the very silly looking query.

      To reproduce, uncomment this line in the test lang/largeCodeGen.java
      // testLogicalOperators(con, 10000);

      FAILED QUERY: Logical operators with 10000 parameters. java.lang.StackOverflowError
      FAILED QUERY: IN clause with 3300 parameters. ERROR 40XT0: An internal error was identified by RawStore module.
      at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:301)
      at org.apache.derby.impl.store.raw.xact.Xact.setActiveState(Xact.java:1770)
      at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1271)
      at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:865)
      at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:614)
      at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:478)
      at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1315)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:7251)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1470)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1380)
      at org.apache.derby.impl.sql.compile.QueryTreeNode.getSchemaDescriptor(QueryTreeNode.java:1460)
      at org.apache.derby.impl.sql.compile.QueryTreeNode.getSchemaDescriptor(QueryTreeNode.java:1412)
      at org.apache.derby.impl.sql.compile.FromBaseTable.bindTableDescriptor(FromBaseTable.java:2370)
      at org.apache.derby.impl.sql.compile.FromBaseTable.bindNonVTITables(FromBaseTable.java:2120)
      at org.apache.derby.impl.sql.compile.FromList.bindTables(FromList.java:307)
      at org.apache.derby.impl.sql.compile.SelectNode.bindNonVTITables(SelectNode.java:434)
      at org.apache.derby.impl.sql.compile.DMLStatementNode.bindTables(DMLStatementNode.java:217)
      at org.apache.derby.impl.sql.compile.DMLStatementNode.bind(DMLStatementNode.java:158)
      at org.apache.derby.impl.sql.compile.ReadCursorNode.bind(ReadCursorNode.java:74)
      at org.apache.derby.impl.sql.compile.CursorNode.bind(CursorNode.java:250)
      at org.apache.derby.impl.sql.GenericStatement.prepMinion(GenericStatement.java:336)
      at org.apache.deERROR 40XT0: An internal error was identified by RawStore module.
      at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:301)
      at org.apache.derby.impl.store.raw.xact.Xact.setActiveState(Xact.java:1770)
      at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1271)
      at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:865)
      at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:614)
      at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:478)
      at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1315)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:7251)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1470)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1380)
      at org.apache.derby.impl.sql.compile.QueryTreeNode.getSchemaDescriptor(QueryTreeNode.java:1460)
      at org.apache.derby.impl.sql.compile.QueryTreeNode.getSchemaDescriptor(QueryTreeNode.java:1412)
      at org.apache.derby.impl.sql.compile.DDLStatementNode.getTableDescriptor(DDLStatementNode.java:263)
      at org.apache.derby.impl.sql.compile.DDLStatementNode.getTableDescriptor(DDLStatementNode.java:234)
      at org.apache.derby.impl.sql.compile.DropTableNode.bind(DropTableNode.java:114)
      at org.apache.derby.impl.sql.GenericStatement.prepMinion(GenericStatement.java:336)
      at org.apache.derby.impl.sql.GenericStatement.prepare(GenericStatement.java:110)
      at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(GenericLanguageConnectionContext.java:704)
      at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:533)
      at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(EmbedStatement.java:158)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.createTestTable(Unknown Source)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.testUnions(Unknown Source)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.main(Unknown Source)
      rby.impl.sql.GenericStatement.prepare(GenericStatement.java:110)
      at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(GenericLanguageConnectionContext.java:704)
      at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(EmbedPreparedStatement.java:118)
      at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(EmbedPreparedStatement20.java:82)
      at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(EmbedPreparedStatement30.java:62)
      at org.apache.derby.jdbc.Driver30.newEmbedPreparedStatement(Driver30.java:92)
      at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(EmbedConnection.java:678)
      at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(EmbedConnection.java:522)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.checkT0Query(Unknown Source)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.testInClause(Unknown Source)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.testInClause(Unknown Source)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.main(Unknown Source)
      Exception in thread "main"

      NOTE:
      I was actually able to connect to the database after the query failed despite the RawStore error so maybe the db is not corrupted but just the transaction state out of whack for the connection. This needs more investigation. I am putting this as SQL for now, even though we don't have a stack trace for the StackOverflow. This is just a guess at component.

      1. run1tst.out
        1.08 MB
        Myrna van Lunteren

        Issue Links

          Activity

          Hide
          Daniel John Debrunner added a comment -

          I think this is a general issue, not specific to code generation. It's similar to DERBY-443, where an OutOfMemoryException would shut the system down. In this case I think the StackOverflowError is shutting Derby down, when we could keep running since the problem is specific to a single thread.

          Show
          Daniel John Debrunner added a comment - I think this is a general issue, not specific to code generation. It's similar to DERBY-443 , where an OutOfMemoryException would shut the system down. In this case I think the StackOverflowError is shutting Derby down, when we could keep running since the problem is specific to a single thread.
          Hide
          Kathey Marsden added a comment -

          The internal error is a manifestation of DERBY-443 Make Derby engine robust in case of OutOfMemoryError exceptions

          Closing this issue out as duplicate.

          Show
          Kathey Marsden added a comment - The internal error is a manifestation of DERBY-443 Make Derby engine robust in case of OutOfMemoryError exceptions Closing this issue out as duplicate.
          Hide
          Daniel John Debrunner added a comment -

          I don't see this as a duplicate of DERBY-443. 443 is specific to OutOfMemoryErrors which are not the same as StackOverflowErrors.
          One aspect of the handling could be similar to 443, but the cause is very different. It might be a programming or algorithm error that leads to a StackOverflowError.

          Show
          Daniel John Debrunner added a comment - I don't see this as a duplicate of DERBY-443 . 443 is specific to OutOfMemoryErrors which are not the same as StackOverflowErrors. One aspect of the handling could be similar to 443, but the cause is very different. It might be a programming or algorithm error that leads to a StackOverflowError.
          Hide
          Rick Hillegas added a comment -

          Triaged for 10.5.3: assigned normal urgency and marked as a crash.

          Show
          Rick Hillegas added a comment - Triaged for 10.5.3: assigned normal urgency and marked as a crash.
          Hide
          Guy Davis added a comment -

          Long SELECT query with many OR clauses still a problem in 10.6. Any ETA on when this will be handled? We run the same size query against our MS SQL Server and Oracle DBs without incident.

          Show
          Guy Davis added a comment - Long SELECT query with many OR clauses still a problem in 10.6. Any ETA on when this will be handled? We run the same size query against our MS SQL Server and Oracle DBs without incident.
          Hide
          Dag H. Wanvik added a comment -

          Trying this now, I see that the generated class for the query code gets too large. From derby.log:

          ERROR XBCM4: Java class file format limit(s) exceeded: method:e1 code_length (65577 > 65535) in generated class org.apache.derby.exe.aceaa980c4x012fx01b8xbd24x000070524bdd3.
          at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:303)
          at org.apache.derby.impl.services.bytecode.BCClass.getClassBytecode(BCClass.java:174)
          at org.apache.derby.impl.services.bytecode.GClass.getGeneratedClass(GClass.java:63)
          at org.apache.derby.impl.sql.compile.ExpressionClassBuilder.getGeneratedClass(ExpressionClassBuilder.java:849)
          at org.apache.derby.impl.sql.compile.StatementNode.generate(StatementNode.java:388)
          at org.apache.derby.impl.sql.GenericStatement.prepMinion(GenericStatement.java:472)
          at org.apache.derby.impl.sql.GenericStatement.prepare(GenericStatement.java:93)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(GenericLanguageConnectionContext.java:1101)
          :

          I do not see a stack overflow error or database corruption (I could connect to via ij). In order to support such large queries, one solution would be to partition up on the generated method "e1" into several methods somehow.

          Show
          Dag H. Wanvik added a comment - Trying this now, I see that the generated class for the query code gets too large. From derby.log: ERROR XBCM4: Java class file format limit(s) exceeded: method:e1 code_length (65577 > 65535) in generated class org.apache.derby.exe.aceaa980c4x012fx01b8xbd24x000070524bdd3. at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:303) at org.apache.derby.impl.services.bytecode.BCClass.getClassBytecode(BCClass.java:174) at org.apache.derby.impl.services.bytecode.GClass.getGeneratedClass(GClass.java:63) at org.apache.derby.impl.sql.compile.ExpressionClassBuilder.getGeneratedClass(ExpressionClassBuilder.java:849) at org.apache.derby.impl.sql.compile.StatementNode.generate(StatementNode.java:388) at org.apache.derby.impl.sql.GenericStatement.prepMinion(GenericStatement.java:472) at org.apache.derby.impl.sql.GenericStatement.prepare(GenericStatement.java:93) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(GenericLanguageConnectionContext.java:1101) : I do not see a stack overflow error or database corruption (I could connect to via ij). In order to support such large queries, one solution would be to partition up on the generated method "e1" into several methods somehow.
          Hide
          Myrna van Lunteren added a comment -

          I tried this, on a Windows 7, with trunk updated to 1511883, using ibm 1.6 SR14, with sane classes, and I did get a stack overflow. I'm attaching the output.

          Show
          Myrna van Lunteren added a comment - I tried this, on a Windows 7, with trunk updated to 1511883, using ibm 1.6 SR14, with sane classes, and I did get a stack overflow. I'm attaching the output.

            People

            • Assignee:
              Unassigned
              Reporter:
              Kathey Marsden
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development