Derby
  1. Derby
  2. DERBY-3698

Bulk SQL INSERT, SELECT, UPDATE operations creates OutOfMemoryError after first Cleanup action on serverside

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Incomplete
    • Affects Version/s: 10.4.1.3
    • Fix Version/s: None
    • Component/s: Network Server
    • Labels:
      None
    • Environment:
      Windows XP, JDK 1.4

      Description

      In the migration of an DB,
      By using JDBC (connection is open one time in the begining)
      after many INSERT, SELECT,UPDATE statement executions (more than 3500)
      I got the following Exception on the client side:

      SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='INSTITUTE OF EDUCATION University OF LONDON'

      com.javaDevHome.ladyBeetle.exception.MicroJDBCException: java.sql.SQLException: DERBY SQL error: SQLCODE: -1, SQLSTATE: XJ001, SQLERRMC: java.lang.OutOfMemoryErrorXJ001.U
      at com.javaDevHome.ladyBeetle.MicroJDBCConnection.executeQuery(MicroJDBCConnection.java:251)
      at com.javaDevHome.ladyBeetle.MicroJDBCDBase.selectRowSilent(MicroJDBCDBase.java:133)
      at euProj.transfer.Transfer.procFirmNew(Transfer.java:259)
      at euProj.transfer.Transfer.procshortListNew(Transfer.java:200)
      at euProj.transfer.Transfer.readConsortiums(Transfer.java:524)
      at euProj.transfer.Transfer.procContract(Transfer.java:107)
      at euProj.transfer.Transfer.readLine(Transfer.java:554)
      at euProj.transfer.Transfer.main(Transfer.java:594)
      Caused by: java.sql.SQLException: DERBY SQL error: SQLCODE: -1, SQLSTATE: XJ001, SQLERRMC: java.lang.OutOfMemoryErrorXJ001.U
      at org.apache.derby.client.am.SQLExceptionFactory40.getSQLException(Unknown Source)
      at org.apache.derby.client.am.SqlException.getSQLException(Unknown Source)
      at org.apache.derby.client.am.Statement.executeQuery(Unknown Source)
      at com.javaDevHome.ladyBeetle.MicroJDBCConnection.executeQuery(MicroJDBCConnection.java:247)
      ... 7 more
      Caused by: org.apache.derby.client.am.SqlException: DERBY SQL error: SQLCODE: -1, SQLSTATE: XJ001, SQLERRMC: java.lang.OutOfMemoryErrorXJ001.U
      at org.apache.derby.client.am.Statement.completeSqlca(Unknown Source)
      at org.apache.derby.client.net.NetStatementReply.parsePrepareError(Unknown Source)
      at org.apache.derby.client.net.NetStatementReply.parsePRPSQLSTTreply(Unknown Source)
      at org.apache.derby.client.net.NetStatementReply.readPrepareDescribeOutput(Unknown Source)
      at org.apache.derby.client.net.StatementReply.readPrepareDescribeOutput(Unknown Source)
      at org.apache.derby.client.net.NetStatement.readPrepareDescribeOutput_(Unknown Source)
      at org.apache.derby.client.am.Statement.readPrepareDescribeOutput(Unknown Source)
      at org.apache.derby.client.am.Statement.flowExecute(Unknown Source)
      at org.apache.derby.client.am.Statement.executeQueryX(Unknown Source)
      ... 9 more

      in derby.log on the server side :

      2008-05-29 20:33:04.578 GMT Thread[DRDAConnThread_5,5,main] (XID = 171959), (SESSIONID = 4), (DATABASE = euProj03), (DRDAID = NF000001.H1CF-4339779978832484505

      {5}), Begin compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='INSTITUTE OF EDUCATION University OF LONDON' :End prepared statement
      2008-05-29 20:33:04.609 GMT Thread[DRDAConnThread_5,5,main] (XID = 171959), (SESSIONID = 4), (DATABASE = euProj03), (DRDAID = NF000001.H1CF-4339779978832484505{5}

      ), End compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='INSTITUTE OF EDUCATION University OF LONDON' :End prepared statement
      2008-05-29 20:33:07.499 GMT Thread[DRDAConnThread_5,5,main] (XID = 171959), (SESSIONID = 4), (DATABASE = euProj03), (DRDAID = NF000001.H1CF-4339779978832484505

      {5}), Cleanup action starting
      2008-05-29 20:33:07.499 GMT Thread[DRDAConnThread_5,5,main] (XID = 171959), (SESSIONID = 4), (DATABASE = euProj03), (DRDAID = NF000001.H1CF-4339779978832484505{5}

      ), Failed Statement is: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='INSTITUTE OF EDUCATION University OF LONDON'
      java.lang.OutOfMemoryError
      Cleanup action completed
      2008-05-29 20:33:37.359 GMT Thread[DRDAConnThread_6,5,main] (XID = 169430), (SESSIONID = 6), (DATABASE = euProj03), (DRDAID = NF000001.H1D1-4255618960795994455

      {7}), Rolling back
      2008-05-29 20:33:37.359 GMT Thread[DRDAConnThread_6,5,main] (XID = 169430), (SESSIONID = 6), (DATABASE = euProj03), (DRDAID = NF000001.H1D1-4255618960795994455{7}

      ), Rolling back
      2008-05-29 20:33:37.359 GMT Thread[DRDAConnThread_4,5,main] (XID = 143043), (SESSIONID = 5), (DATABASE = euProj03), (DRDAID = NF000001.H1D0-4255900435772705908

      {6}), Rolling back
      2008-05-29 20:33:37.359 GMT Thread[DRDAConnThread_4,5,main] (XID = 143043), (SESSIONID = 5), (DATABASE = euProj03), (DRDAID = NF000001.H1D0-4255900435772705908{6}

      ), Rolling back

      in the derby.log there is no other Cleanup action, this is the first and the only one.

      I have started the server by default heap memory options:
      java -Dderby.system.home=C:\derby -jar %DERBY_HOME%\lib\derbyrun.jar server start

      1. ac004b0019x011ax5fe9x6337x0000760c8b634386.class
        5 kB
        Semih ARAL
      2. ac38c30028x011ax550cx96f4x0000760c8b6335b.class
        5 kB
        Semih ARAL
      3. ac80220011x011ax5a87xb7a0x0000760c8b6345b3.class
        3 kB
        Semih ARAL
      4. ac80220011x011ax5c5ax34c5x0000760c8b636b66.class
        7 kB
        Semih ARAL
      5. ac80220011x011ax550cx96f4x0000760c8b63674e.class
        3 kB
        Semih ARAL
      6. ac80220011x011ax620ax9a0bx0000760c8b63574a.class
        5 kB
        Semih ARAL
      7. ac80220011x011ax5218x3922x0000760c8b635ec2.class
        3 kB
        Semih ARAL
      8. derby01.log
        1 kB
        Semih ARAL
      9. derby02.log
        0.9 kB
        Semih ARAL
      10. derby03.log
        1 kB
        Semih ARAL
      11. derby04.log
        0.1 kB
        Semih ARAL
      12. log.txt
        11 kB
        Semih ARAL

        Activity

        Hide
        Knut Anders Hatlen added a comment -

        Triaged for 10.5.2. No response from the reporter to the request for more information, so marking the issue as incomplete.

        Show
        Knut Anders Hatlen added a comment - Triaged for 10.5.2. No response from the reporter to the request for more information, so marking the issue as incomplete.
        Hide
        Kathey Marsden added a comment -

        Could you possibly post a reproduction for this issue or more information to help us reproduce? If not I think we should close this as CannotReproduce.

        Thanks

        Kathey

        Show
        Kathey Marsden added a comment - Could you possibly post a reproduction for this issue or more information to help us reproduce? If not I think we should close this as CannotReproduce. Thanks Kathey
        Hide
        Kristian Waagan added a comment -

        I see the following statement in an earlier comment:
        "So if one use jdbc connect/disconnect for each operation, after several minutes, derby easily rejects a new connection, because it is busy."

        Although one can argue this approach should work, it is very inefficient to create a new connection for each SQL query.
        As mentioned before, using prepared statements is also crucial to get good performance.

        It would be interesting to know how you obtain the connections. Did you use DriverManager or a DataSource implementation?
        Unless your program ends up keeping a lot of connections open, Derby should not run out of memory. If it does, it indicates a bug.

        Show
        Kristian Waagan added a comment - I see the following statement in an earlier comment: "So if one use jdbc connect/disconnect for each operation, after several minutes, derby easily rejects a new connection, because it is busy." Although one can argue this approach should work, it is very inefficient to create a new connection for each SQL query. As mentioned before, using prepared statements is also crucial to get good performance. It would be interesting to know how you obtain the connections. Did you use DriverManager or a DataSource implementation? Unless your program ends up keeping a lot of connections open, Derby should not run out of memory. If it does, it indicates a bug.
        Hide
        Rick Hillegas added a comment -

        Here are two other approaches for speeding up your migration:

        1) Take advantage of JDBC statement batching (see PreparedStatement.addBatch())

        2) Use a table function to feed data to Derby. Here you would wrap your source data stream in a table function and then issue a query like the following:

        insert into targetTable
        select * from table( myTableFunction( ... ) ) s

        For an example of how to do this, please see the demoForeignDbmsVtis.sql script in the demo subtree of the 10.4 Derby distribution. Look for the countryLanguage table function.

        Show
        Rick Hillegas added a comment - Here are two other approaches for speeding up your migration: 1) Take advantage of JDBC statement batching (see PreparedStatement.addBatch()) 2) Use a table function to feed data to Derby. Here you would wrap your source data stream in a table function and then issue a query like the following: insert into targetTable select * from table( myTableFunction( ... ) ) s For an example of how to do this, please see the demoForeignDbmsVtis.sql script in the demo subtree of the 10.4 Derby distribution. Look for the countryLanguage table function.
        Hide
        Semih ARAL added a comment -

        The migration was terminated. The data were transferred to derby DB with a new structure and constraints.

        I write this comment to share my experience with derby DB users and developers.

        In bulk operations derby reacts well in the beginning, but after several minutes it slows down.
        So if one use jdbc connect/disconnect for each operation, after several minutes, derby easily rejects a new connection, because it is busy.
        This stops the bulk operations, so the connection must be open one time and must be kept open till the end of the operations.

        Derby has a problem of memory management for bulk operations, after 5-10 minutes it gives OutOfemory error:
        At first from the derby.log, I guessed that there is a cleanup process which creates OutOfMemory error, I had a look at the code, and saw that the word cleanup appears just in the log, I was not sure about the relation between cleanup and OutOfMemory error.
        To prevent OutOfMemory error for my special case, I added System.gc() in the begining of GenericStatement.prepMinion on derby server, of course this slowed down the derby, but time to get OutOfMemory was prolonged enough.

        During the migration I got several exceptions on the server side and derby created some class files, I don't know these files are useful or not but to help the derby developers I will put all them to attachments.

        For derby users, in bulk operations I advice to use import/export functions if it is possible, because I haven't seen any OutOfMemory error in import/export of very big tables.

        Good luck.

        Show
        Semih ARAL added a comment - The migration was terminated. The data were transferred to derby DB with a new structure and constraints. I write this comment to share my experience with derby DB users and developers. In bulk operations derby reacts well in the beginning, but after several minutes it slows down. So if one use jdbc connect/disconnect for each operation, after several minutes, derby easily rejects a new connection, because it is busy. This stops the bulk operations, so the connection must be open one time and must be kept open till the end of the operations. Derby has a problem of memory management for bulk operations, after 5-10 minutes it gives OutOfemory error: At first from the derby.log, I guessed that there is a cleanup process which creates OutOfMemory error, I had a look at the code, and saw that the word cleanup appears just in the log, I was not sure about the relation between cleanup and OutOfMemory error. To prevent OutOfMemory error for my special case, I added System.gc() in the begining of GenericStatement.prepMinion on derby server, of course this slowed down the derby, but time to get OutOfMemory was prolonged enough. During the migration I got several exceptions on the server side and derby created some class files, I don't know these files are useful or not but to help the derby developers I will put all them to attachments. For derby users, in bulk operations I advice to use import/export functions if it is possible, because I haven't seen any OutOfMemory error in import/export of very big tables. Good luck.
        Hide
        Semih ARAL added a comment -

        I will try for heap dump, but it wil not be very different than the followng I think :
        http://mail-archives.apache.org/mod_mbox/db-derby-dev/200702.mbox/%3C7632076.1171898345554.JavaMail.jira@brutus%3E

        I tried to execute our program with derby.language.logStatementText=false, nothing changed:

        Apache Derby Network Server - 10.4.1.3 - (648739) started and ready to accept connections on port 1527 at 2008-05-31 14:27:31.514 GMT
        ----------------------------------------------------------------
        2008-05-31 14:27:36.639 GMT:
        Booting Derby version The Apache Software Foundation - Apache Derby - 10.4.1.3 - (648739): instance c013800d-011a-3f60-45fb-0000760c8b63
        on database directory C:\home\semih\project\mask\euProjects\data\openOfficeDb\derby\euProj02

        Database Class Loader started - derby.database.classpath=''
        2008-05-31 14:39:48.217 GMT Thread[DRDAConnThread_3,5,main] (XID = 3393258), (SESSIONID = 2), (DATABASE = euProj02), (DRDAID = NF000001.H172-580400189733304267

        {3}), Cleanup action starting
        2008-05-31 14:39:48.217 GMT Thread[DRDAConnThread_3,5,main] (XID = 3393258), (SESSIONID = 2), (DATABASE = euProj02), (DRDAID = NF000001.H172-580400189733304267{3}

        ), Failed Statement is: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='Universite Libre de Bruxeels (BE)'
        java.lang.OutOfMemoryError
        Cleanup action completed

        As you see I got this exception after almost 10 minutes of execution and claculate that the migration/restructuring will take approximately 16 hours, after that point I will not try any proposition concerning less memory usage.

        I think that if cleanup creates OutOfMemoryError, instead of releasing the memory, the problem is there.

        Show
        Semih ARAL added a comment - I will try for heap dump, but it wil not be very different than the followng I think : http://mail-archives.apache.org/mod_mbox/db-derby-dev/200702.mbox/%3C7632076.1171898345554.JavaMail.jira@brutus%3E I tried to execute our program with derby.language.logStatementText=false, nothing changed: Apache Derby Network Server - 10.4.1.3 - (648739) started and ready to accept connections on port 1527 at 2008-05-31 14:27:31.514 GMT ---------------------------------------------------------------- 2008-05-31 14:27:36.639 GMT: Booting Derby version The Apache Software Foundation - Apache Derby - 10.4.1.3 - (648739): instance c013800d-011a-3f60-45fb-0000760c8b63 on database directory C:\home\semih\project\mask\euProjects\data\openOfficeDb\derby\euProj02 Database Class Loader started - derby.database.classpath='' 2008-05-31 14:39:48.217 GMT Thread [DRDAConnThread_3,5,main] (XID = 3393258), (SESSIONID = 2), (DATABASE = euProj02), (DRDAID = NF000001.H172-580400189733304267 {3}), Cleanup action starting 2008-05-31 14:39:48.217 GMT Thread [DRDAConnThread_3,5,main] (XID = 3393258), (SESSIONID = 2), (DATABASE = euProj02), (DRDAID = NF000001.H172-580400189733304267{3} ), Failed Statement is: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='Universite Libre de Bruxeels (BE)' java.lang.OutOfMemoryError Cleanup action completed As you see I got this exception after almost 10 minutes of execution and claculate that the migration/restructuring will take approximately 16 hours, after that point I will not try any proposition concerning less memory usage. I think that if cleanup creates OutOfMemoryError, instead of releasing the memory, the problem is there.
        Hide
        Kristian Waagan added a comment -

        Any chance you can take a heap dump and post the histogram (from the server side) when or just before the error happens?
        I don't remember the specifics for Java 1.4, but it is easy to do with Java SE 6 at least.
        I'm assuming this data and/or application isn't available to the public?

        I also have two more questions;

        • Have you tried doing the migration using prepared statements?
        • Have you set derby.language.logStatementText=true ? If so, have you tried without setting it?
        Show
        Kristian Waagan added a comment - Any chance you can take a heap dump and post the histogram (from the server side) when or just before the error happens? I don't remember the specifics for Java 1.4, but it is easy to do with Java SE 6 at least. I'm assuming this data and/or application isn't available to the public? I also have two more questions; Have you tried doing the migration using prepared statements? Have you set derby.language.logStatementText=true ? If so, have you tried without setting it?
        Hide
        Semih ARAL added a comment -

        I commited periodically, the first cleanup action provoked the OutOfMemoryError again.

        derby.log

        2008-05-31 12:35:33.576 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235

        {2}), Begin compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='PG Finances (FR)' :End prepared statement
        2008-05-31 12:35:33.592 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2}

        ), End compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='PG Finances (FR)' :End prepared statement
        2008-05-31 12:35:33.592 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235

        {2}), Executing prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='PG Finances (FR)' :End prepared statement
        2008-05-31 12:35:33.623 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2}

        ), Begin compiling prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8282, 100, 5, 691, 0) :End prepared statement
        2008-05-31 12:35:34.780 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235

        {2}), End compiling prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8282, 100, 5, 691, 0) :End prepared statement
        2008-05-31 12:35:34.780 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2}

        ), Executing prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8282, 100, 5, 691, 0) :End prepared statement
        2008-05-31 12:35:34.780 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235

        {2}), Committing
        2008-05-31 12:35:34.811 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2}

        ), Begin compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='PRIVATA' :End prepared statement
        2008-05-31 12:35:34.826 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235

        {2}), End compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='PRIVATA' :End prepared statement
        2008-05-31 12:35:34.842 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2}

        ), Executing prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='PRIVATA' :End prepared statement
        2008-05-31 12:35:34.858 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235

        {2}), Begin compiling prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8283, 100, 5, 692, 0) :End prepared statement
        2008-05-31 12:35:34.858 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2}

        ), End compiling prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8283, 100, 5, 692, 0) :End prepared statement
        2008-05-31 12:35:34.873 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235

        {2}), Executing prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8283, 100, 5, 692, 0) :End prepared statement
        2008-05-31 12:35:34.873 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2}

        ), Committing
        2008-05-31 12:35:34.905 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235

        {2}), Begin compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='SEQUA Foundation for Economic Development and Vocational Training (DE)' :End prepared statement
        2008-05-31 12:35:34.936 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2}

        ), End compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='SEQUA Foundation for Economic Development and Vocational Training (DE)' :End prepared statement
        2008-05-31 12:35:34.936 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235

        {2}), Executing prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='SEQUA Foundation for Economic Development and Vocational Training (DE)' :End prepared statement
        2008-05-31 12:35:34.951 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2}

        ), Begin compiling prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8284, 100, 5, 693, 0) :End prepared statement
        2008-05-31 12:35:34.967 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235

        {2}), End compiling prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8284, 100, 5, 693, 0) :End prepared statement
        2008-05-31 12:35:37.389 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2}

        ), Cleanup action starting
        2008-05-31 12:35:37.389 GMT Thread[DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235

        {2}

        ), Failed Statement is: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8284, 100, 5, 693, 0)
        java.lang.OutOfMemoryError
        Cleanup action completed
        2008-05-31 12:36:02.967 GMT Thread[DRDAConnThread_4,5,main] (XID = 3293757), (SESSIONID = 3), (DATABASE = euProj02), (DRDAID = NF000001.GCA9-4469539942952112219

        {4}), Rolling back
        2008-05-31 12:36:02.967 GMT Thread[DRDAConnThread_4,5,main] (XID = 3293757), (SESSIONID = 3), (DATABASE = euProj02), (DRDAID = NF000001.GCA9-4469539942952112219{4}

        ), Rolling back
        2008-05-31 12:36:02.967 GMT Thread[DRDAConnThread_2,5,main] (XID = 3267370), (SESSIONID = 2), (DATABASE = euProj02), (DRDAID = NF000001.GCA8-4469821417928824563

        {3}), Rolling back
        2008-05-31 12:36:02.967 GMT Thread[DRDAConnThread_2,5,main] (XID = 3267370), (SESSIONID = 2), (DATABASE = euProj02), (DRDAID = NF000001.GCA8-4469821417928824563{3}

        ), Rolling back

        Show
        Semih ARAL added a comment - I commited periodically, the first cleanup action provoked the OutOfMemoryError again. derby.log 2008-05-31 12:35:33.576 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235 {2}), Begin compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='PG Finances (FR)' :End prepared statement 2008-05-31 12:35:33.592 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2} ), End compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='PG Finances (FR)' :End prepared statement 2008-05-31 12:35:33.592 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235 {2}), Executing prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='PG Finances (FR)' :End prepared statement 2008-05-31 12:35:33.623 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2} ), Begin compiling prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8282, 100, 5, 691, 0) :End prepared statement 2008-05-31 12:35:34.780 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235 {2}), End compiling prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8282, 100, 5, 691, 0) :End prepared statement 2008-05-31 12:35:34.780 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2} ), Executing prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8282, 100, 5, 691, 0) :End prepared statement 2008-05-31 12:35:34.780 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296419), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235 {2}), Committing 2008-05-31 12:35:34.811 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2} ), Begin compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='PRIVATA' :End prepared statement 2008-05-31 12:35:34.826 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235 {2}), End compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='PRIVATA' :End prepared statement 2008-05-31 12:35:34.842 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2} ), Executing prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='PRIVATA' :End prepared statement 2008-05-31 12:35:34.858 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235 {2}), Begin compiling prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8283, 100, 5, 692, 0) :End prepared statement 2008-05-31 12:35:34.858 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2} ), End compiling prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8283, 100, 5, 692, 0) :End prepared statement 2008-05-31 12:35:34.873 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235 {2}), Executing prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8283, 100, 5, 692, 0) :End prepared statement 2008-05-31 12:35:34.873 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296422), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2} ), Committing 2008-05-31 12:35:34.905 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235 {2}), Begin compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='SEQUA Foundation for Economic Development and Vocational Training (DE)' :End prepared statement 2008-05-31 12:35:34.936 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2} ), End compiling prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='SEQUA Foundation for Economic Development and Vocational Training (DE)' :End prepared statement 2008-05-31 12:35:34.936 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235 {2}), Executing prepared statement: SELECT PK_SEQ, NAME, URL FROM EUPROJ.FIRM WHERE NAME ='SEQUA Foundation for Economic Development and Vocational Training (DE)' :End prepared statement 2008-05-31 12:35:34.951 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2} ), Begin compiling prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8284, 100, 5, 693, 0) :End prepared statement 2008-05-31 12:35:34.967 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235 {2}), End compiling prepared statement: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8284, 100, 5, 693, 0) :End prepared statement 2008-05-31 12:35:37.389 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235{2} ), Cleanup action starting 2008-05-31 12:35:37.389 GMT Thread [DRDAConnThread_3,5,main] (XID = 3296425), (SESSIONID = 1), (DATABASE = euProj02), (DRDAID = NF000001.GCA7-4470102892905539235 {2} ), Failed Statement is: INSERT INTO EUPROJ.SHORTLIST (PK_SEQ, FK_LOT_SEQ, CONSORTIUM_NO, FK_FIRM_SEQ, LEAD) VALUES (8284, 100, 5, 693, 0) java.lang.OutOfMemoryError Cleanup action completed 2008-05-31 12:36:02.967 GMT Thread [DRDAConnThread_4,5,main] (XID = 3293757), (SESSIONID = 3), (DATABASE = euProj02), (DRDAID = NF000001.GCA9-4469539942952112219 {4}), Rolling back 2008-05-31 12:36:02.967 GMT Thread [DRDAConnThread_4,5,main] (XID = 3293757), (SESSIONID = 3), (DATABASE = euProj02), (DRDAID = NF000001.GCA9-4469539942952112219{4} ), Rolling back 2008-05-31 12:36:02.967 GMT Thread [DRDAConnThread_2,5,main] (XID = 3267370), (SESSIONID = 2), (DATABASE = euProj02), (DRDAID = NF000001.GCA8-4469821417928824563 {3}), Rolling back 2008-05-31 12:36:02.967 GMT Thread [DRDAConnThread_2,5,main] (XID = 3267370), (SESSIONID = 2), (DATABASE = euProj02), (DRDAID = NF000001.GCA8-4469821417928824563{3} ), Rolling back
        Hide
        Semih ARAL added a comment -

        I am using default JDBC autocommit mode, but I will try to send periodic commit statements.
        I should say that problem is deterministic, in bulk operations, first cleanup action provokes OutOfMemoryError.

        Show
        Semih ARAL added a comment - I am using default JDBC autocommit mode, but I will try to send periodic commit statements. I should say that problem is deterministic, in bulk operations, first cleanup action provokes OutOfMemoryError.
        Hide
        Kathey Marsden added a comment -

        You might try a periodic commit to reduce memory usage.

        Show
        Kathey Marsden added a comment - You might try a periodic commit to reduce memory usage.

          People

          • Assignee:
            Unassigned
            Reporter:
            Semih ARAL
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development