Derby
  1. Derby
  2. DERBY-759

Query with thousands of logical operators and parameters causes StackOverflowError and then ERROR 40XT0: An internal error was identified by RawStore module on next statement

    Details

    • Urgency:
      Normal
    • Bug behavior facts:
      Crash

      Description

      An SQL Statement with thousands of logical operators and parameters causes a StackOverflow Error and then the next statement throws a ERROR 40XT0: An internal error was identified by RawStore module.

      The statement is of the form SELECT * FROM T0 WHERE (si = ? AND si = ? ) OR (si = ? AND si = ? ) ...

      I came across this issue when looking at DERBY-739 and code generation issues for large queries, thus the very silly looking query.

      To reproduce, uncomment this line in the test lang/largeCodeGen.java
      // testLogicalOperators(con, 10000);

      FAILED QUERY: Logical operators with 10000 parameters. java.lang.StackOverflowError
      FAILED QUERY: IN clause with 3300 parameters. ERROR 40XT0: An internal error was identified by RawStore module.
      at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:301)
      at org.apache.derby.impl.store.raw.xact.Xact.setActiveState(Xact.java:1770)
      at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1271)
      at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:865)
      at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:614)
      at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:478)
      at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1315)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:7251)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1470)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1380)
      at org.apache.derby.impl.sql.compile.QueryTreeNode.getSchemaDescriptor(QueryTreeNode.java:1460)
      at org.apache.derby.impl.sql.compile.QueryTreeNode.getSchemaDescriptor(QueryTreeNode.java:1412)
      at org.apache.derby.impl.sql.compile.FromBaseTable.bindTableDescriptor(FromBaseTable.java:2370)
      at org.apache.derby.impl.sql.compile.FromBaseTable.bindNonVTITables(FromBaseTable.java:2120)
      at org.apache.derby.impl.sql.compile.FromList.bindTables(FromList.java:307)
      at org.apache.derby.impl.sql.compile.SelectNode.bindNonVTITables(SelectNode.java:434)
      at org.apache.derby.impl.sql.compile.DMLStatementNode.bindTables(DMLStatementNode.java:217)
      at org.apache.derby.impl.sql.compile.DMLStatementNode.bind(DMLStatementNode.java:158)
      at org.apache.derby.impl.sql.compile.ReadCursorNode.bind(ReadCursorNode.java:74)
      at org.apache.derby.impl.sql.compile.CursorNode.bind(CursorNode.java:250)
      at org.apache.derby.impl.sql.GenericStatement.prepMinion(GenericStatement.java:336)
      at org.apache.deERROR 40XT0: An internal error was identified by RawStore module.
      at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:301)
      at org.apache.derby.impl.store.raw.xact.Xact.setActiveState(Xact.java:1770)
      at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1271)
      at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:865)
      at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:614)
      at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:478)
      at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1315)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:7251)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1470)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1380)
      at org.apache.derby.impl.sql.compile.QueryTreeNode.getSchemaDescriptor(QueryTreeNode.java:1460)
      at org.apache.derby.impl.sql.compile.QueryTreeNode.getSchemaDescriptor(QueryTreeNode.java:1412)
      at org.apache.derby.impl.sql.compile.DDLStatementNode.getTableDescriptor(DDLStatementNode.java:263)
      at org.apache.derby.impl.sql.compile.DDLStatementNode.getTableDescriptor(DDLStatementNode.java:234)
      at org.apache.derby.impl.sql.compile.DropTableNode.bind(DropTableNode.java:114)
      at org.apache.derby.impl.sql.GenericStatement.prepMinion(GenericStatement.java:336)
      at org.apache.derby.impl.sql.GenericStatement.prepare(GenericStatement.java:110)
      at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(GenericLanguageConnectionContext.java:704)
      at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:533)
      at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(EmbedStatement.java:158)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.createTestTable(Unknown Source)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.testUnions(Unknown Source)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.main(Unknown Source)
      rby.impl.sql.GenericStatement.prepare(GenericStatement.java:110)
      at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(GenericLanguageConnectionContext.java:704)
      at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(EmbedPreparedStatement.java:118)
      at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(EmbedPreparedStatement20.java:82)
      at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(EmbedPreparedStatement30.java:62)
      at org.apache.derby.jdbc.Driver30.newEmbedPreparedStatement(Driver30.java:92)
      at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(EmbedConnection.java:678)
      at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(EmbedConnection.java:522)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.checkT0Query(Unknown Source)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.testInClause(Unknown Source)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.testInClause(Unknown Source)
      at org.apache.derbyTesting.functionTests.tests.lang.largeCodeGen.main(Unknown Source)
      Exception in thread "main"

      NOTE:
      I was actually able to connect to the database after the query failed despite the RawStore error so maybe the db is not corrupted but just the transaction state out of whack for the connection. This needs more investigation. I am putting this as SQL for now, even though we don't have a stack trace for the StackOverflow. This is just a guess at component.

      1. run1tst.out
        1.08 MB
        Myrna van Lunteren

        Issue Links

          Activity

          Kathey Marsden created issue -
          Hide
          Daniel John Debrunner added a comment -

          I think this is a general issue, not specific to code generation. It's similar to DERBY-443, where an OutOfMemoryException would shut the system down. In this case I think the StackOverflowError is shutting Derby down, when we could keep running since the problem is specific to a single thread.

          Show
          Daniel John Debrunner added a comment - I think this is a general issue, not specific to code generation. It's similar to DERBY-443 , where an OutOfMemoryException would shut the system down. In this case I think the StackOverflowError is shutting Derby down, when we could keep running since the problem is specific to a single thread.
          Hide
          Kathey Marsden added a comment -

          The internal error is a manifestation of DERBY-443 Make Derby engine robust in case of OutOfMemoryError exceptions

          Closing this issue out as duplicate.

          Show
          Kathey Marsden added a comment - The internal error is a manifestation of DERBY-443 Make Derby engine robust in case of OutOfMemoryError exceptions Closing this issue out as duplicate.
          Kathey Marsden made changes -
          Field Original Value New Value
          Resolution Duplicate [ 3 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Kathey Marsden made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Assignee Kathey Marsden [ kmarsden ]
          Hide
          Daniel John Debrunner added a comment -

          I don't see this as a duplicate of DERBY-443. 443 is specific to OutOfMemoryErrors which are not the same as StackOverflowErrors.
          One aspect of the handling could be similar to 443, but the cause is very different. It might be a programming or algorithm error that leads to a StackOverflowError.

          Show
          Daniel John Debrunner added a comment - I don't see this as a duplicate of DERBY-443 . 443 is specific to OutOfMemoryErrors which are not the same as StackOverflowErrors. One aspect of the handling could be similar to 443, but the cause is very different. It might be a programming or algorithm error that leads to a StackOverflowError.
          Daniel John Debrunner made changes -
          Resolution Duplicate [ 3 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Kathey Marsden made changes -
          Assignee Kathey Marsden [ kmarsden ]
          A B made changes -
          Link This issue relates to DERBY-1818 [ DERBY-1818 ]
          Hide
          Rick Hillegas added a comment -

          Triaged for 10.5.3: assigned normal urgency and marked as a crash.

          Show
          Rick Hillegas added a comment - Triaged for 10.5.3: assigned normal urgency and marked as a crash.
          Rick Hillegas made changes -
          Bug behavior facts [Crash]
          Urgency Normal
          Kathey Marsden made changes -
          Labels derby_triage10_5_2
          Hide
          Guy Davis added a comment -

          Long SELECT query with many OR clauses still a problem in 10.6. Any ETA on when this will be handled? We run the same size query against our MS SQL Server and Oracle DBs without incident.

          Show
          Guy Davis added a comment - Long SELECT query with many OR clauses still a problem in 10.6. Any ETA on when this will be handled? We run the same size query against our MS SQL Server and Oracle DBs without incident.
          Hide
          Dag H. Wanvik added a comment -

          Trying this now, I see that the generated class for the query code gets too large. From derby.log:

          ERROR XBCM4: Java class file format limit(s) exceeded: method:e1 code_length (65577 > 65535) in generated class org.apache.derby.exe.aceaa980c4x012fx01b8xbd24x000070524bdd3.
          at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:303)
          at org.apache.derby.impl.services.bytecode.BCClass.getClassBytecode(BCClass.java:174)
          at org.apache.derby.impl.services.bytecode.GClass.getGeneratedClass(GClass.java:63)
          at org.apache.derby.impl.sql.compile.ExpressionClassBuilder.getGeneratedClass(ExpressionClassBuilder.java:849)
          at org.apache.derby.impl.sql.compile.StatementNode.generate(StatementNode.java:388)
          at org.apache.derby.impl.sql.GenericStatement.prepMinion(GenericStatement.java:472)
          at org.apache.derby.impl.sql.GenericStatement.prepare(GenericStatement.java:93)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(GenericLanguageConnectionContext.java:1101)
          :

          I do not see a stack overflow error or database corruption (I could connect to via ij). In order to support such large queries, one solution would be to partition up on the generated method "e1" into several methods somehow.

          Show
          Dag H. Wanvik added a comment - Trying this now, I see that the generated class for the query code gets too large. From derby.log: ERROR XBCM4: Java class file format limit(s) exceeded: method:e1 code_length (65577 > 65535) in generated class org.apache.derby.exe.aceaa980c4x012fx01b8xbd24x000070524bdd3. at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:303) at org.apache.derby.impl.services.bytecode.BCClass.getClassBytecode(BCClass.java:174) at org.apache.derby.impl.services.bytecode.GClass.getGeneratedClass(GClass.java:63) at org.apache.derby.impl.sql.compile.ExpressionClassBuilder.getGeneratedClass(ExpressionClassBuilder.java:849) at org.apache.derby.impl.sql.compile.StatementNode.generate(StatementNode.java:388) at org.apache.derby.impl.sql.GenericStatement.prepMinion(GenericStatement.java:472) at org.apache.derby.impl.sql.GenericStatement.prepare(GenericStatement.java:93) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(GenericLanguageConnectionContext.java:1101) : I do not see a stack overflow error or database corruption (I could connect to via ij). In order to support such large queries, one solution would be to partition up on the generated method "e1" into several methods somehow.
          Gavin made changes -
          Workflow jira [ 12344851 ] Default workflow, editable Closed status [ 12801365 ]
          Hide
          Myrna van Lunteren added a comment -

          I tried this, on a Windows 7, with trunk updated to 1511883, using ibm 1.6 SR14, with sane classes, and I did get a stack overflow. I'm attaching the output.

          Show
          Myrna van Lunteren added a comment - I tried this, on a Windows 7, with trunk updated to 1511883, using ibm 1.6 SR14, with sane classes, and I did get a stack overflow. I'm attaching the output.
          Myrna van Lunteren made changes -
          Labels derby_triage10_5_2 derby_triage10_11 derby_triage10_5_2
          Myrna van Lunteren made changes -
          Attachment run1tst.out [ 12597589 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          70d 9h 47m 1 Kathey Marsden 24/Feb/06 08:56
          Resolved Resolved Closed Closed
          30s 1 Kathey Marsden 24/Feb/06 08:57
          Closed Closed Reopened Reopened
          45d 21h 46m 1 Daniel John Debrunner 11/Apr/06 07:43

            People

            • Assignee:
              Unassigned
              Reporter:
              Kathey Marsden
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development