Derby
  1. Derby
  2. DERBY-4075

ERROR XSDBB: Unknown page format at page Page(613,Container(0, 1024)) when running MailJdbc (Embedded) system tests

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Duplicate
    • Affects Version/s: 10.5.1.1
    • Fix Version/s: 10.5.1.1
    • Component/s: Store
    • Labels:
      None
    • Environment:
      windows 2000 professional, ibm 1.6 (sr2), revision 10.5: 745360
    • Urgency:
      Urgent
    • Issue & fix info:
      High Value Fix
    • Bug behavior facts:
      Data corruption, Regression

      Description

      I started the org.apache.derbyTesting.system.mailJdbc Embedded test and after about 1 day it ran into an apparently corrupted database.

      2009-02-19 14:02:44.221 GMT Thread[Refresh Thread,5,main] (XID = 349909), (SESSIONID = 1), (DATABASE = mailsdb), (DRDAID = null), Failed Statement is: insert into REFRESH.INBOX(from_name,to_name,date,Message,attach_id,size_problem) values (?,?,?,?,?,?) with 6 parameters begin parameter #1: ABCE :end parameter begin parameter #2: WXYY :end parameter begin parameter #3: 2009-02-19 06:02:43.705 :end parameter begin parameter #4: CLOB(org.apache.derby.iapi.types.ReaderToUTF8Stream@41ba41ba) :end parameter begin parameter #5: 0 :end parameter begin parameter #6: This column is used only to by pass the space problem. If the problem still exists, then we are going to have a serious issue here.***************************************************************************************************** :end parameter
      ERROR XSDBB: Unknown page format at page Page(613,Container(0, 1024)), page dump follows: Hex dump:

      The error up to that point appear to me no different from other errors that this test is expected to produce (40001, 23505, 4XL01) except that the very first error is a grant error.

      I think we can't have a release until this corruption is explained away or fixed.

      1. windowsfail3.zip
        9.22 MB
        Myrna van Lunteren
      2. windowsfail1.zip
        5.16 MB
        Myrna van Lunteren
      3. windows5logout.jar
        327 kB
        Myrna van Lunteren
      4. performance.out
        221 kB
        Myrna van Lunteren
      5. linuxfail1.jar
        6.12 MB
        Myrna van Lunteren
      6. jdkrc0winmailfail.jar
        9.46 MB
        Myrna van Lunteren
      7. derbylog.zip
        26 kB
        Myrna van Lunteren
      8. derby.log.run2
        849 kB
        Myrna van Lunteren
      9. d4075_debugchange.diff
        0.9 kB
        Myrna van Lunteren
      10. Activity.out
        2.95 MB
        Myrna van Lunteren

        Issue Links

          Activity

          Hide
          Myrna van Lunteren added a comment -

          attaching a zipped deryb.log

          Show
          Myrna van Lunteren added a comment - attaching a zipped deryb.log
          Hide
          Mike Matrigali added a comment -

          I went through the derby.log - do continue to run this against a SANE server as we are getting
          more information into derby.log that way.

          I assume the db is too big to post to the jira. Can you try to connect to it with IJ and see if it
          boots ok? It would be interesting to know if the problem was in memory or on disk.

          Can you post any more information about the environment? Number of processors, cores per processor, whether disk write cache is enabled or not, what kind of disks? Just want to get a
          feel on how concurrent the test is being.

          Does this test log what it has done in any other way, maybe some sort of test specific log? Off hand do you know if compress table is being tested? The last thing in the log is a failed backup, which I think can be ignored as I think it failed because we set the db corrupt by the previous error. Did the test do any valid backups before this? I think this test has uncovered
          I/O coordination problems between backup and the main code before.

          It looks like we got an exception from an initPage() call on Page(613,Container(0, 1024)). From the hex dump it looks like the page is all 0's. A free page should not be all 0's it should have a valid page header at least. Only un-allocated page

          From a little bit above this we dumped all the alloc page information on the table. Specifics are there but of interest is that pages 1 through 612 are in use, and 613 - 813 free, 814-1050 mostly in use (a few scattered free page), and finally 1050-86672 are unallocated - I think this is just what the debugging prints for the remaining bits of any alloc page it dumps.

          Show
          Mike Matrigali added a comment - I went through the derby.log - do continue to run this against a SANE server as we are getting more information into derby.log that way. I assume the db is too big to post to the jira. Can you try to connect to it with IJ and see if it boots ok? It would be interesting to know if the problem was in memory or on disk. Can you post any more information about the environment? Number of processors, cores per processor, whether disk write cache is enabled or not, what kind of disks? Just want to get a feel on how concurrent the test is being. Does this test log what it has done in any other way, maybe some sort of test specific log? Off hand do you know if compress table is being tested? The last thing in the log is a failed backup, which I think can be ignored as I think it failed because we set the db corrupt by the previous error. Did the test do any valid backups before this? I think this test has uncovered I/O coordination problems between backup and the main code before. It looks like we got an exception from an initPage() call on Page(613,Container(0, 1024)). From the hex dump it looks like the page is all 0's. A free page should not be all 0's it should have a valid page header at least. Only un-allocated page From a little bit above this we dumped all the alloc page information on the table. Specifics are there but of interest is that pages 1 through 612 are in use, and 613 - 813 free, 814-1050 mostly in use (a few scattered free page), and finally 1050-86672 are unallocated - I think this is just what the debugging prints for the remaining bits of any alloc page it dumps.
          Hide
          Myrna van Lunteren added a comment -

          I'm attaching the performance.out and Activity.out files that the test generates. From the look of this, it seems multiple successful backups including compressesook place.

          Show
          Myrna van Lunteren added a comment - I'm attaching the performance.out and Activity.out files that the test generates. From the look of this, it seems multiple successful backups including compressesook place.
          Hide
          Myrna van Lunteren added a comment -

          This is a 2790 MHZ cpu, 1000 MB hd, 4000 MB ram, 32-bit, IBM , X-Series 8676 machine.
          I believe it's a 1CPU machine that has hyperthreading switched on.
          The device manager indicates the disk is a IBM-ESXS SCSI - is that the info you need?
          There was a couple of G of disk space left.
          Right now I can't check if disk write cache is enabled, or what kind of disks are in use, or verify that this is a 1CPU with hyperthreading vs. 2 CPU machine, but I'll try to get that info.

          I've kicked off the test (after saving the old testdir) again, will report if the issue reproduces...

          Show
          Myrna van Lunteren added a comment - This is a 2790 MHZ cpu, 1000 MB hd, 4000 MB ram, 32-bit, IBM , X-Series 8676 machine. I believe it's a 1CPU machine that has hyperthreading switched on. The device manager indicates the disk is a IBM-ESXS SCSI - is that the info you need? There was a couple of G of disk space left. Right now I can't check if disk write cache is enabled, or what kind of disks are in use, or verify that this is a 1CPU with hyperthreading vs. 2 CPU machine, but I'll try to get that info. I've kicked off the test (after saving the old testdir) again, will report if the issue reproduces...
          Hide
          Myrna van Lunteren added a comment -

          And yes, I can connect to the db after the error with ij...

          Show
          Myrna van Lunteren added a comment - And yes, I can connect to the db after the error with ij...
          Hide
          Kristian Waagan added a comment -

          I kicked off the test too, using two different machines. One is a CMT machine (32 execution threads, low CPU frequency), the other is a 2xdual core machine (2.8 GHz). Both are running Solaris and Java SE 6.
          I'll keep them running for a while and grep the logs for XSDBB.

          Show
          Kristian Waagan added a comment - I kicked off the test too, using two different machines. One is a CMT machine (32 execution threads, low CPU frequency), the other is a 2xdual core machine (2.8 GHz). Both are running Solaris and Java SE 6. I'll keep them running for a while and grep the logs for XSDBB.
          Hide
          Myrna van Lunteren added a comment -

          I didn't (and don't) have much time today, but I took a quick look at my run and it's not run into the XSDBB, but instead, I see this:

          ERROR 38000: The exception 'java.sql.SQLException: Java exception: 'ASSERT FAILED got null page following long column chain. Head column piece at Page(106,Container(0, 1024)) null page at 296: org.apache.derby.shared.common.sanity.AssertFailure'.' was thrown while evaluating an expression.

          Also an unexpected error for this test.

          Show
          Myrna van Lunteren added a comment - I didn't (and don't) have much time today, but I took a quick look at my run and it's not run into the XSDBB, but instead, I see this: ERROR 38000: The exception 'java.sql.SQLException: Java exception: 'ASSERT FAILED got null page following long column chain. Head column piece at Page(106,Container(0, 1024)) null page at 296: org.apache.derby.shared.common.sanity.AssertFailure'.' was thrown while evaluating an expression. Also an unexpected error for this test.
          Hide
          Kristian Waagan added a comment -

          My three instances of the test have been running since the 27th of February. I haven't observed any of the problems mentioned in this Jira issue.
          I'm running on Solaris 10, using Sun JDK 1.6.0_06-b02.

          However, I have some other questions / observations:
          a) Are you running with SQL authorization disabled?
          I'm getting errors when the backup is performed:
          : ERROR :Backup Thread : Error while doing the Compress procedure: The exception
          'java.sql.SQLException: User 'BACKUP' can not perform the operation in schema
          'REFRESH'.' was thrown while evaluating an expression.

          b) Is the test supposed to work with the network driver?
          I'm getter error messages about accessing a LOB after it has been freed.

          Based on the answers to these questions, I can reconfigure and restart the tests.

          Show
          Kristian Waagan added a comment - My three instances of the test have been running since the 27th of February. I haven't observed any of the problems mentioned in this Jira issue. I'm running on Solaris 10, using Sun JDK 1.6.0_06-b02. However, I have some other questions / observations: a) Are you running with SQL authorization disabled? I'm getting errors when the backup is performed: : ERROR :Backup Thread : Error while doing the Compress procedure: The exception 'java.sql.SQLException: User 'BACKUP' can not perform the operation in schema 'REFRESH'.' was thrown while evaluating an expression. b) Is the test supposed to work with the network driver? I'm getter error messages about accessing a LOB after it has been freed. Based on the answers to these questions, I can reconfigure and restart the tests.
          Hide
          Myrna van Lunteren added a comment -

          b) As far as I understand the test, if you start it with the parameter 'NetworkServer' it should run with network server. But I ran - so far - with embedded.

          a) I started it with no specific settings of any kind.

          I don't see those backup, or compress errors, but I do see a grant error pop up as the first thing, which I'm not sure is supposed to happen:

          "2009-03-02 19:26:26.516 GMT Thread[Refresh Thread,5,main] (XID = 210), (SESSIONID = 1), (DATABASE = mailsdb), (DRDAID = null), Failed Statement is: grant select on REFRESH.INBOX to BROWSE
          ERROR 42Z60: GRANT not allowed unless database property derby.database.sqlAuthorization has value 'TRUE'."

          I had to restart the test again (the machine was rebooted) on Monday, and I do see the XSDBB error again - after about 1 day...

          Show
          Myrna van Lunteren added a comment - b) As far as I understand the test, if you start it with the parameter 'NetworkServer' it should run with network server. But I ran - so far - with embedded. a) I started it with no specific settings of any kind. I don't see those backup, or compress errors, but I do see a grant error pop up as the first thing, which I'm not sure is supposed to happen: "2009-03-02 19:26:26.516 GMT Thread [Refresh Thread,5,main] (XID = 210), (SESSIONID = 1), (DATABASE = mailsdb), (DRDAID = null), Failed Statement is: grant select on REFRESH.INBOX to BROWSE ERROR 42Z60: GRANT not allowed unless database property derby.database.sqlAuthorization has value 'TRUE'." I had to restart the test again (the machine was rebooted) on Monday, and I do see the XSDBB error again - after about 1 day...
          Hide
          Myrna van Lunteren added a comment -

          oops - the derby.properties file somehow was lost in the shuffle. I'm rerunning with it in place.

          Show
          Myrna van Lunteren added a comment - oops - the derby.properties file somehow was lost in the shuffle. I'm rerunning with it in place.
          Hide
          Kristian Waagan added a comment -

          I'm not sure which JVMs will work on which platforms etc, but are you able to run with a different JVM, or at least a different version, to rule out a JVM bug?
          Also, are you seeing the error on several machines?

          My test runs still haven't produced any of the page format errors, but I haven't run them without SQL authorization yet. If the SQL authorization is causing the backup to not be run, that may be significant.
          Another idea is to increase the load of the test and see if the error occurs sooner then (I don't know how to actually increase the load, but assume there are some parameters we can adjust).

          It turns out one of my tests ran out of disk space. The database had grown to 25 GB, which seems to be a lot larger than on the other machine I'm running on.
          I don't know what the expected database growth rate for this test is.

          Show
          Kristian Waagan added a comment - I'm not sure which JVMs will work on which platforms etc, but are you able to run with a different JVM, or at least a different version, to rule out a JVM bug? Also, are you seeing the error on several machines? My test runs still haven't produced any of the page format errors, but I haven't run them without SQL authorization yet. If the SQL authorization is causing the backup to not be run, that may be significant. Another idea is to increase the load of the test and see if the error occurs sooner then (I don't know how to actually increase the load, but assume there are some parameters we can adjust). It turns out one of my tests ran out of disk space. The database had grown to 25 GB, which seems to be a lot larger than on the other machine I'm running on. I don't know what the expected database growth rate for this test is.
          Hide
          Myrna van Lunteren added a comment -

          With the derby.properties in place, the error did not occur.
          That doesn't mean there isn't a problem, but I'll need to look much closer at the test to see what it's doing if the properties are not in place. If the test ends up doing a double boot of the database, this is known to be not supported and cause corruption.

          So, I'll

          • start the test with 10.4 without derby.properties and see if the problem occurs with that version - if so, then this is not a regression.
          • verify the concistency of the database and backed up database of the run in which the error occurred with the consistency checker.
          • try to figure out what the test is doing when the properties it expects are not in place.
          Show
          Myrna van Lunteren added a comment - With the derby.properties in place, the error did not occur. That doesn't mean there isn't a problem, but I'll need to look much closer at the test to see what it's doing if the properties are not in place. If the test ends up doing a double boot of the database, this is known to be not supported and cause corruption. So, I'll start the test with 10.4 without derby.properties and see if the problem occurs with that version - if so, then this is not a regression. verify the concistency of the database and backed up database of the run in which the error occurred with the consistency checker. try to figure out what the test is doing when the properties it expects are not in place.
          Hide
          Kristian Waagan added a comment -

          My tests have been running since the 27th, 5th and 6th. The database sizes vary from 7.2 GB to 11 GB.

          I have seen the following errors in derby.log (merged output from all three runs, grep error | sort -u, removed one caused by):
          ERROR 23505: The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index identified by 'ATTACH__PK' defined on 'ATTACH'.
          ERROR 38000: The exception 'java.sql.SQLException: User 'BACKUP' can not perform the operation in schema 'REFRESH'.' was thrown while evaluating an expression.
          ERROR 40001: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
          ERROR 40XL1: A lock could not be obtained within the time requested
          ERROR 42504: User 'BACKUP' does not have execute permission on PROCEDURE 'SYSCS_UTIL'.'SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE_NOWAIT'.
          ERROR 42507: User 'BACKUP' can not perform the operation in schema 'REFRESH'.
          ERROR 42Z60: GRANT not allowed unless database property derby.database.sqlAuthorization has value 'TRUE'.

          I'll restart the two newer tests, removing derby.properties.

          Show
          Kristian Waagan added a comment - My tests have been running since the 27th, 5th and 6th. The database sizes vary from 7.2 GB to 11 GB. I have seen the following errors in derby.log (merged output from all three runs, grep error | sort -u, removed one caused by): ERROR 23505: The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index identified by 'ATTACH__PK' defined on 'ATTACH'. ERROR 38000: The exception 'java.sql.SQLException: User 'BACKUP' can not perform the operation in schema 'REFRESH'.' was thrown while evaluating an expression. ERROR 40001: A lock could not be obtained due to a deadlock, cycle of locks and waiters is: ERROR 40XL1: A lock could not be obtained within the time requested ERROR 42504: User 'BACKUP' does not have execute permission on PROCEDURE 'SYSCS_UTIL'.'SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE_NOWAIT'. ERROR 42507: User 'BACKUP' can not perform the operation in schema 'REFRESH'. ERROR 42Z60: GRANT not allowed unless database property derby.database.sqlAuthorization has value 'TRUE'. I'll restart the two newer tests, removing derby.properties.
          Hide
          Myrna van Lunteren added a comment -

          I ran with 10.4.2.0 on the same machine, with the same jvm, and without derby.properties and so far the problem has not manifested. Which seems to point to a regression.
          With 10.5 the problem showed up (1st with XSDBB, 2nd 38000, 3rd run XSDBB) after about 1 day. That suggests it is related to the backup process, which according to the notes in the test will occur after about 1 day.

          I've kicked off another run on a non-windows box, same jvm version (ibm16) without derby.properties, to see what that does.

          I couldn't check the consistency of the tables (yet)...although I could start ij and even select from a table, I could not do the select suggested on the page: http://wiki.apache.org/db-derby/DatabaseConsistencyCheck. (got the XSDBB error right away).

          Show
          Myrna van Lunteren added a comment - I ran with 10.4.2.0 on the same machine, with the same jvm, and without derby.properties and so far the problem has not manifested. Which seems to point to a regression. With 10.5 the problem showed up (1st with XSDBB, 2nd 38000, 3rd run XSDBB) after about 1 day. That suggests it is related to the backup process, which according to the notes in the test will occur after about 1 day. I've kicked off another run on a non-windows box, same jvm version (ibm16) without derby.properties, to see what that does. I couldn't check the consistency of the tables (yet)...although I could start ij and even select from a table, I could not do the select suggested on the page: http://wiki.apache.org/db-derby/DatabaseConsistencyCheck . (got the XSDBB error right away).
          Hide
          Myrna van Lunteren added a comment -

          attaching a small patch which prints out a timestamp into the Activity.out and performance.out files, so as to get a better idea of what's happening when.

          Show
          Myrna van Lunteren added a comment - attaching a small patch which prints out a timestamp into the Activity.out and performance.out files, so as to get a better idea of what's happening when.
          Hide
          Myrna van Lunteren added a comment -

          I noticed that my 10.4 run was with insane jars, and my 10.5 run with sane jars.
          Also, I could not see in the code that the backup happens once a day; it's more like every 2 minutes. Because of the lack of derby.properties the other calls did not get reported (in my 10.4/insane jars run nothing at all showed in derby.log). So, my earlier assumption that the backup was only happening once a day is wrong; the note in the Readme.txt is stale. The fact that the problem occurred after about a day is a coincidence.

          Code inspection suggested a couple of ways to force this more often;

          • cause more thread.interrupts of the refresh thread
          • sleep less

          Also interesting experiments (especially if one of the above mechanisms can make it happen more easily): To see if the problem reproduces

          • on another windows machine
          • with 10.4 and insane jars
          • without compress
          • without backup altogether (but with some other select that would check all tables)
          Show
          Myrna van Lunteren added a comment - I noticed that my 10.4 run was with insane jars, and my 10.5 run with sane jars. Also, I could not see in the code that the backup happens once a day; it's more like every 2 minutes. Because of the lack of derby.properties the other calls did not get reported (in my 10.4/insane jars run nothing at all showed in derby.log). So, my earlier assumption that the backup was only happening once a day is wrong; the note in the Readme.txt is stale. The fact that the problem occurred after about a day is a coincidence. Code inspection suggested a couple of ways to force this more often; cause more thread.interrupts of the refresh thread sleep less Also interesting experiments (especially if one of the above mechanisms can make it happen more easily): To see if the problem reproduces on another windows machine with 10.4 and insane jars without compress without backup altogether (but with some other select that would check all tables)
          Hide
          Myrna van Lunteren added a comment -

          The run on linux failed after about 15 hours with some corruption occurring:

          • ERROR XSLAD: log Record at instant 6,347,963,620,272 in log file 1,478 corrupted
            . Expected log record length 0, real length 32,709.
            This run I ran without derby.properties, so the derby.log only has errors, not the full statement text, but there are these details around the time of the error:

          last error recorded before the XSLAD and around it:
          ---------------------------------------------------------
          2009-03-12 08:59:51.164 GMT Thread[Refresh Thread,5,main] (XID = 373637), (SESSIONID = 1), (DATABASE = mailsdb), (DRDAID = null), Failed Statement is: insert into REFRESH.ATTACH (id,attach_id,attachment) values (?,?,?) with 3 parameters begin parameter #1: 3505 :end parameter begin parameter #2: 5 :end parameter begin parameter #3: BLOB(org.apache.derby.iapi.types.RawToBinaryFormatStream@2a062a06)
          :end parameter
          ERROR 23505: The statement was aborted because it would have caused a duplicate
          key value in a unique or primary key constraint or unique index identified by 'ATTACH__PK' defined on 'ATTACH'.
          at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:303)
          at org.apache.derby.impl.sql.execute.IndexChanger.insertAndCheckDups(IndexChanger.java:439)
          at org.apache.derby.impl.sql.execute.IndexChanger.doInsert(IndexChanger.java:383)
          at org.apache.derby.impl.sql.execute.IndexChanger.insert(IndexChanger.java:589)
          at org.apache.derby.impl.sql.execute.IndexSetChanger.insert(IndexSetChanger.java:267)
          at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:453)
          at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022)
          at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495)
          at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416)
          at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1648)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdateEmbedPreparedStatement.java:294)
          at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.insertMail(DbTasks.java:396)
          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.insertMail(Refresh.java:99)
          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.doWork(Refresh.java:75)
          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.run(Refresh.java:51)

          ------------ BEGIN SHUTDOWN ERROR STACK -------------

          ERROR XSLAD: log Record at instant 6,347,963,620,272 in log file 1,478 corrupted. Expected log record length 0, real length 32,709.
          at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:373)
          at org.apache.derby.impl.store.raw.log.Scan.getNextRecordBackward(Scan.java:376)
          at org.apache.derby.impl.store.raw.log.Scan.getNextRecord(Scan.java:204)
          at org.apache.derby.impl.store.raw.log.FileLogger.undo(FileLogger.java:939)
          at org.apache.derby.impl.store.raw.xact.Xact.popSavePoints(Xact.java:2209)
          at org.apache.derby.impl.store.raw.xact.Xact.rollbackToSavePoint(Xact.java:1562)
          at org.apache.derby.impl.store.access.RAMTransaction.rollbackToSavePoint(RAMTransaction.java:2022)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.internalRollbackToSavepoint(GenericLanguageConnectionContext.java:1512)
          at org.apache.derby.impl.sql.conn.GenericStatementContext.cleanupOnError(GenericStatementContext.java:578)
          at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(ContextManager.java:333)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.cleanupOnError(TransactionResourceImpl.java:419)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleExceptionTransactionResourceImpl.java:337)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2201)
          at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:81)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1323)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1648)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:294)
          at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.insertMail(DbTasks.java:396)
          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.insertMail(Refresh.java:99)
          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.doWork(Refresh.java:75)
          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.run(Refresh.java:51)

          ------------ END SHUTDOWN ERROR STACK -------------

          New exception raised during cleanup Cannot rollback transaction 373637, trying to compensate Page Operation: Page(23062,Container(0, 1056)) pageVersion 10 : Update Slot=0 recordId=8 operation with null
          ERROR XSLA8: Cannot rollback transaction 373637, trying to compensate Page Operation: Page(23062,Container(0, 1056)) pageVersion 10 : Update Slot=0 recordId=operation with null
          at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:366)
          at org.apache.derby.impl.store.raw.log.FileLogger.undo(FileLogger.java:1046)
          at org.apache.derby.impl.store.raw.xact.Xact.popSavePoints(Xact.java:2209)
          at org.apache.derby.impl.store.raw.xact.Xact.rollbackToSavePoint(Xact.java:1562)
          at org.apache.derby.impl.store.access.RAMTransaction.rollbackToSavePoint(RAMTransaction.java:2022)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.internalRollbackToSavepoint(GenericLanguageConnectionContext.java:1512)
          at org.apache.derby.impl.sql.conn.GenericStatementContext.cleanupOnError
          (GenericStatementContext.java:578)
          at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(ContextManager.java:333)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.cleanupOnError(TransactionResourceImpl.java:419)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleExceptionTransactionResourceImpl.java:337)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2201)
          at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:81)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1323)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1648)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:294)
          at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.insertMail(DbTasks.java:396)
          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.insertMail(Refresh.java:99)
          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.doWork(Refresh.java:75)
          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.run(Refresh.java:51)
          ============= end nested exception, level (1) ===========

          2009-03-12 08:59:52.752 GMT:
          Shutting down instance a816c00e-011f-f6a2-11a8-000000097630
          ----------------------------------------------------------------
          2009-03-12 08:59:52.753 GMT Thread[Refresh Thread,5,main] Less severe exception raised during cleanup (ignored) An attempt was made to close a transaction that was still active. The transaction has been aborted.
          ERROR 40XT4: An attempt was made to close a transaction that was still active. The transaction has been aborted.
          at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:276)
          at org.apache.derby.impl.store.raw.xact.Xact.close(Xact.java:1136)
          at org.apache.derby.impl.store.raw.xact.XactContext.cleanupOnError(XactContext.java:140)
          at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(ContextManager.java:333)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.cleanupOnError(TransactionResourceImpl.java:419)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:337)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2201)
          at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:81)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1323)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1648)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:294)
          at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.insertMail(DbTasks.java:396)
          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.insertMail(Refresh.java:99)
          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.doWork(Refresh.java:75)
          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.run(Refresh.java:51)
          Cleanup action completed

          Show
          Myrna van Lunteren added a comment - The run on linux failed after about 15 hours with some corruption occurring: ERROR XSLAD: log Record at instant 6,347,963,620,272 in log file 1,478 corrupted . Expected log record length 0, real length 32,709. This run I ran without derby.properties, so the derby.log only has errors, not the full statement text, but there are these details around the time of the error: last error recorded before the XSLAD and around it: --------------------------------------------------------- 2009-03-12 08:59:51.164 GMT Thread [Refresh Thread,5,main] (XID = 373637), (SESSIONID = 1), (DATABASE = mailsdb), (DRDAID = null), Failed Statement is: insert into REFRESH.ATTACH (id,attach_id,attachment) values (?,?,?) with 3 parameters begin parameter #1: 3505 :end parameter begin parameter #2: 5 :end parameter begin parameter #3: BLOB(org.apache.derby.iapi.types.RawToBinaryFormatStream@2a062a06) :end parameter ERROR 23505: The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index identified by 'ATTACH__PK' defined on 'ATTACH'. at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:303) at org.apache.derby.impl.sql.execute.IndexChanger.insertAndCheckDups(IndexChanger.java:439) at org.apache.derby.impl.sql.execute.IndexChanger.doInsert(IndexChanger.java:383) at org.apache.derby.impl.sql.execute.IndexChanger.insert(IndexChanger.java:589) at org.apache.derby.impl.sql.execute.IndexSetChanger.insert(IndexSetChanger.java:267) at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:453) at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022) at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495) at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416) at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1648) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdateEmbedPreparedStatement.java:294) at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.insertMail(DbTasks.java:396) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.insertMail(Refresh.java:99) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.doWork(Refresh.java:75) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.run(Refresh.java:51) ------------ BEGIN SHUTDOWN ERROR STACK ------------- ERROR XSLAD: log Record at instant 6,347,963,620,272 in log file 1,478 corrupted. Expected log record length 0, real length 32,709. at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:373) at org.apache.derby.impl.store.raw.log.Scan.getNextRecordBackward(Scan.java:376) at org.apache.derby.impl.store.raw.log.Scan.getNextRecord(Scan.java:204) at org.apache.derby.impl.store.raw.log.FileLogger.undo(FileLogger.java:939) at org.apache.derby.impl.store.raw.xact.Xact.popSavePoints(Xact.java:2209) at org.apache.derby.impl.store.raw.xact.Xact.rollbackToSavePoint(Xact.java:1562) at org.apache.derby.impl.store.access.RAMTransaction.rollbackToSavePoint(RAMTransaction.java:2022) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.internalRollbackToSavepoint(GenericLanguageConnectionContext.java:1512) at org.apache.derby.impl.sql.conn.GenericStatementContext.cleanupOnError(GenericStatementContext.java:578) at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(ContextManager.java:333) at org.apache.derby.impl.jdbc.TransactionResourceImpl.cleanupOnError(TransactionResourceImpl.java:419) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleExceptionTransactionResourceImpl.java:337) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2201) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:81) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1323) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1648) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:294) at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.insertMail(DbTasks.java:396) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.insertMail(Refresh.java:99) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.doWork(Refresh.java:75) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.run(Refresh.java:51) ------------ END SHUTDOWN ERROR STACK ------------- New exception raised during cleanup Cannot rollback transaction 373637, trying to compensate Page Operation: Page(23062,Container(0, 1056)) pageVersion 10 : Update Slot=0 recordId=8 operation with null ERROR XSLA8: Cannot rollback transaction 373637, trying to compensate Page Operation: Page(23062,Container(0, 1056)) pageVersion 10 : Update Slot=0 recordId=operation with null at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:366) at org.apache.derby.impl.store.raw.log.FileLogger.undo(FileLogger.java:1046) at org.apache.derby.impl.store.raw.xact.Xact.popSavePoints(Xact.java:2209) at org.apache.derby.impl.store.raw.xact.Xact.rollbackToSavePoint(Xact.java:1562) at org.apache.derby.impl.store.access.RAMTransaction.rollbackToSavePoint(RAMTransaction.java:2022) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.internalRollbackToSavepoint(GenericLanguageConnectionContext.java:1512) at org.apache.derby.impl.sql.conn.GenericStatementContext.cleanupOnError (GenericStatementContext.java:578) at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(ContextManager.java:333) at org.apache.derby.impl.jdbc.TransactionResourceImpl.cleanupOnError(TransactionResourceImpl.java:419) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleExceptionTransactionResourceImpl.java:337) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2201) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:81) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1323) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1648) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:294) at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.insertMail(DbTasks.java:396) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.insertMail(Refresh.java:99) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.doWork(Refresh.java:75) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.run(Refresh.java:51) ============= end nested exception, level (1) =========== 2009-03-12 08:59:52.752 GMT: Shutting down instance a816c00e-011f-f6a2-11a8-000000097630 ---------------------------------------------------------------- 2009-03-12 08:59:52.753 GMT Thread [Refresh Thread,5,main] Less severe exception raised during cleanup (ignored) An attempt was made to close a transaction that was still active. The transaction has been aborted. ERROR 40XT4: An attempt was made to close a transaction that was still active. The transaction has been aborted. at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:276) at org.apache.derby.impl.store.raw.xact.Xact.close(Xact.java:1136) at org.apache.derby.impl.store.raw.xact.XactContext.cleanupOnError(XactContext.java:140) at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(ContextManager.java:333) at org.apache.derby.impl.jdbc.TransactionResourceImpl.cleanupOnError(TransactionResourceImpl.java:419) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:337) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2201) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:81) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1323) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1648) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:294) at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.insertMail(DbTasks.java:396) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.insertMail(Refresh.java:99) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.doWork(Refresh.java:75) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.run(Refresh.java:51) Cleanup action completed
          Hide
          Kristian Waagan added a comment -

          Sorry for asking again Myrna, but can you summarize the hardware (including number of CPUs and features like HyperThreading), platforms and JVM versions you have seen the error on?

          I keep running the tests, and they do not fail. Maybe someone else can spare some cycles and run the tests too?

          I have been running with revision 748421 on Solaris and Linux, but with and without the derby.properties file, using the following two JVMs:

          o java version "1.6.0_06"
          Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
          Java HotSpot(TM) Server VM (build 10.0-b22, mixed mode)

          o java version "1.6.0"
          Java(TM) SE Runtime Environment (build pxi3260sr3-20081106_07(SR3))
          IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260-20081105_25433 (JIT enabled, AOT enabled)
          J9VM - 20081105_025433_lHdSMr
          JIT - r9_20081031_1330
          GC - 20081027_AB)
          JCL - 20081106_01

          Show
          Kristian Waagan added a comment - Sorry for asking again Myrna, but can you summarize the hardware (including number of CPUs and features like HyperThreading), platforms and JVM versions you have seen the error on? I keep running the tests, and they do not fail. Maybe someone else can spare some cycles and run the tests too? I have been running with revision 748421 on Solaris and Linux, but with and without the derby.properties file, using the following two JVMs: o java version "1.6.0_06" Java(TM) SE Runtime Environment (build 1.6.0_06-b02) Java HotSpot(TM) Server VM (build 10.0-b22, mixed mode) o java version "1.6.0" Java(TM) SE Runtime Environment (build pxi3260sr3-20081106_07(SR3)) IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260-20081105_25433 (JIT enabled, AOT enabled) J9VM - 20081105_025433_lHdSMr JIT - r9_20081031_1330 GC - 20081027_AB) JCL - 20081106_01
          Hide
          Kathey Marsden added a comment -

          Well I am not sure if this is worth reporting or if it just muddies the issue. I ran this test for 9 hours on my Windows XP 2002 Service pack , dual core Intel CPU T2600, 2.16GHz machine with
          java version "1.6.0"
          Java(TM) SE Runtime Environment (build pwi3260sr3-20081106_07(SR3))
          IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260-20081105_25433 (JIT enabled, AOT enabled)
          J9VM - 20081105_025433_lHdSMr
          JIT - r9_20081031_1330
          GC - 20081027_AB)
          JCL - 20081106_01

          I had the following in my derby.properties
          derby.language.logStatementText=true
          derby.infolog.append=true
          derby.stream.error.logSeverityLevel=0

          I had changed the sleeps in the tests to be 1/10 of their original value.
          After nine hours it complained that it was out of disk space, but actually I had about 6GB left, so I am not sure why that happened. I didn't see any corruption though.

          ERROR XSLA4: Cannot write to the log, most likely the log is full. Please delete unnecessary files. It is also possible that the file system is read only, or the disk has failed, or some other problems with the media.

          at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:279)

          at org.apache.derby.impl.store.raw.log.LogToFile.appendLogRecord(LogToFile.java:3800)

          at org.apache.derby.impl.store.raw.log.FileLogger.logAndDo(FileLogger.java:370)

          at org.apache.derby.impl.store.raw.xact.Xact.logAndDo(Xact.java:1223)

          at org.apache.derby.impl.store.raw.data.LoggableActions.doAction(LoggableActions.java:221)

          at org.apache.derby.impl.store.raw.data.LoggableActions.actionInsert(LoggableActions.java:143)

          at org.apache.derby.impl.store.raw.data.BasePage.insertLongColumn(BasePage.java:897)

          at org.apache.derby.impl.store.raw.data.BasePage.insertAllowOverflow(BasePage.java:784)

          at org.apache.derby.impl.store.raw.data.BasePage.insert(BasePage.java:653)

          at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:307)

          at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599)

          at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452)

          at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022)

          at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495)

          at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416)

          at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297)

          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235)

          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1648)

          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:294)

          at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.insertMail(DbTasks.java:396)

          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.insertMail(Refresh.java:99)

          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.doWork(Refresh.java:75)

          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.run(Refresh.java:51)

          Caused by: java.io.IOException: There is not enough space on the disk.

          at java.io.RandomAccessFile.writeBytes(Native Method)

          at java.io.RandomAccessFile.write(RandomAccessFile.java:477)

          at org.apache.derby.impl.store.raw.log.LogAccessFile.writeToLog(LogAccessFile.java:785)

          at org.apache.derby.impl.store.raw.log.LogAccessFile.flushDirtyBuffers(LogAccessFile.java:521)

          at org.apache.derby.impl.store.raw.log.LogAccessFile.switchLogBuffer(LogAccessFile.java:611)

          at org.apache.derby.impl.store.raw.log.LogAccessFile.reserveSpaceForChecksum(LogAccessFile.java:855)

          at org.apache.derby.impl.store.raw.log.LogToFile.appendLogRecord(LogToFile.java:3766)

          ... 21 more

          ============= begin nested exception, level (1) ===========

          java.io.IOException: There is not enough space on the disk.

          at java.io.RandomAccessFile.writeBytes(Native Method)

          at java.io.RandomAccessFile.write(RandomAccessFile.java:477)

          at org.apache.derby.impl.store.raw.log.LogAccessFile.writeToLog(LogAccessFile.java:785)

          at org.apache.derby.impl.store.raw.log.LogAccessFile.flushDirtyBuffers(LogAccessFile.java:521)

          at org.apache.derby.impl.store.raw.log.LogAccessFile.switchLogBuffer(LogAccessFile.java:611)

          at org.apache.derby.impl.store.raw.log.LogAccessFile.reserveSpaceForChecksum(LogAccessFile.java:855)

          at org.apache.derby.impl.store.raw.log.LogToFile.appendLogRecord(LogToFile.java:3766)

          at org.apache.derby.impl.store.raw.log.FileLogger.logAndDo(FileLogger.java:370)

          at org.apache.derby.impl.store.raw.xact.Xact.logAndDo(Xact.java:1223)

          at org.apache.derby.impl.store.raw.data.LoggableActions.doAction(LoggableActions.java:221)

          at org.apache.derby.impl.store.raw.data.LoggableActions.actionInsert(LoggableActions.java:143)

          at org.apache.derby.impl.store.raw.data.BasePage.insertLongColumn(BasePage.java:897)

          at org.apache.derby.impl.store.raw.data.BasePage.insertAllowOverflow(BasePage.java:784)

          at org.apache.derby.impl.store.raw.data.BasePage.insert(BasePage.java:653)

          at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:307)

          at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599)

          at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452)

          at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022)

          at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495)

          at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416)

          at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297)

          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235)

          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1648)

          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:294)

          at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.insertMail(DbTasks.java:396)

          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.insertMail(Refresh.java:99)

          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.doWork(Refresh.java:75)

          at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.run(Refresh.java:51)

          ============= end nested exception, level (1) ===========

          Show
          Kathey Marsden added a comment - Well I am not sure if this is worth reporting or if it just muddies the issue. I ran this test for 9 hours on my Windows XP 2002 Service pack , dual core Intel CPU T2600, 2.16GHz machine with java version "1.6.0" Java(TM) SE Runtime Environment (build pwi3260sr3-20081106_07(SR3)) IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260-20081105_25433 (JIT enabled, AOT enabled) J9VM - 20081105_025433_lHdSMr JIT - r9_20081031_1330 GC - 20081027_AB) JCL - 20081106_01 I had the following in my derby.properties derby.language.logStatementText=true derby.infolog.append=true derby.stream.error.logSeverityLevel=0 I had changed the sleeps in the tests to be 1/10 of their original value. After nine hours it complained that it was out of disk space, but actually I had about 6GB left, so I am not sure why that happened. I didn't see any corruption though. ERROR XSLA4: Cannot write to the log, most likely the log is full. Please delete unnecessary files. It is also possible that the file system is read only, or the disk has failed, or some other problems with the media. at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:279) at org.apache.derby.impl.store.raw.log.LogToFile.appendLogRecord(LogToFile.java:3800) at org.apache.derby.impl.store.raw.log.FileLogger.logAndDo(FileLogger.java:370) at org.apache.derby.impl.store.raw.xact.Xact.logAndDo(Xact.java:1223) at org.apache.derby.impl.store.raw.data.LoggableActions.doAction(LoggableActions.java:221) at org.apache.derby.impl.store.raw.data.LoggableActions.actionInsert(LoggableActions.java:143) at org.apache.derby.impl.store.raw.data.BasePage.insertLongColumn(BasePage.java:897) at org.apache.derby.impl.store.raw.data.BasePage.insertAllowOverflow(BasePage.java:784) at org.apache.derby.impl.store.raw.data.BasePage.insert(BasePage.java:653) at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:307) at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599) at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452) at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022) at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495) at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416) at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1648) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:294) at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.insertMail(DbTasks.java:396) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.insertMail(Refresh.java:99) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.doWork(Refresh.java:75) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.run(Refresh.java:51) Caused by: java.io.IOException: There is not enough space on the disk. at java.io.RandomAccessFile.writeBytes(Native Method) at java.io.RandomAccessFile.write(RandomAccessFile.java:477) at org.apache.derby.impl.store.raw.log.LogAccessFile.writeToLog(LogAccessFile.java:785) at org.apache.derby.impl.store.raw.log.LogAccessFile.flushDirtyBuffers(LogAccessFile.java:521) at org.apache.derby.impl.store.raw.log.LogAccessFile.switchLogBuffer(LogAccessFile.java:611) at org.apache.derby.impl.store.raw.log.LogAccessFile.reserveSpaceForChecksum(LogAccessFile.java:855) at org.apache.derby.impl.store.raw.log.LogToFile.appendLogRecord(LogToFile.java:3766) ... 21 more ============= begin nested exception, level (1) =========== java.io.IOException: There is not enough space on the disk. at java.io.RandomAccessFile.writeBytes(Native Method) at java.io.RandomAccessFile.write(RandomAccessFile.java:477) at org.apache.derby.impl.store.raw.log.LogAccessFile.writeToLog(LogAccessFile.java:785) at org.apache.derby.impl.store.raw.log.LogAccessFile.flushDirtyBuffers(LogAccessFile.java:521) at org.apache.derby.impl.store.raw.log.LogAccessFile.switchLogBuffer(LogAccessFile.java:611) at org.apache.derby.impl.store.raw.log.LogAccessFile.reserveSpaceForChecksum(LogAccessFile.java:855) at org.apache.derby.impl.store.raw.log.LogToFile.appendLogRecord(LogToFile.java:3766) at org.apache.derby.impl.store.raw.log.FileLogger.logAndDo(FileLogger.java:370) at org.apache.derby.impl.store.raw.xact.Xact.logAndDo(Xact.java:1223) at org.apache.derby.impl.store.raw.data.LoggableActions.doAction(LoggableActions.java:221) at org.apache.derby.impl.store.raw.data.LoggableActions.actionInsert(LoggableActions.java:143) at org.apache.derby.impl.store.raw.data.BasePage.insertLongColumn(BasePage.java:897) at org.apache.derby.impl.store.raw.data.BasePage.insertAllowOverflow(BasePage.java:784) at org.apache.derby.impl.store.raw.data.BasePage.insert(BasePage.java:653) at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:307) at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599) at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452) at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022) at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495) at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416) at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1648) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:294) at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.insertMail(DbTasks.java:396) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.insertMail(Refresh.java:99) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.doWork(Refresh.java:75) at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.run(Refresh.java:51) ============= end nested exception, level (1) ===========
          Hide
          Myrna van Lunteren added a comment -

          Thank you, Kristian and Kathey, for running the test...

          My latest attempts have shown no unexpected errors except the GRANT.
          This setup different from the original in that

          • I replace the derbyTesting.jar with a more recent one that also had the extra time printing (as per my patch)
          • added derby.properties:
            derby.stream.error.logSeverityLevel=0
            derby.language.logStatementText=true
            Apparently, one of these differences makes the problem disappear.

          I've also looked at the databases of the runs where there was some problem, and in 2 cases I had no problem selecting from any of the tables (user tables and system tables), and in the 2 cases where I got the XSDBB error once the corrupted table was REFRESH.INBOX and the other REFRESH.ATTACH. In each case the backed up database allowed select on all the tables.

          I've run now on 2 machines. The failing situations were on:

          • windows machine:
            OS: Microsoft Windows 2000, service Pack 4
            java: java version "1.6.0"
            Java(TM) SE Runtime Environment (build pwi3260sr2-20080607_01(SR2))
            IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows 2000 x86-32 jvmwi3260-20080603_19964 (JIT enabled, AOT enabled)
            J9VM - 20080603_019964_lHdSMr
            JIT - r9_20080602_1330
            GC - 20080603_AB)
            JCL - 20080606_02
            IBM eserver, Two cpu's, x86 Family 15 Model 2 Stepping 7 GenuiniteIntel ~2790 Mhz, no hyperthreading
            No Write cache
            One core per cpu
            drives: 1 36gb scsi 1 72gbscsi (I'm running on the 62gb drive).

          Linux:
          Suse 9.1 (I know, oldish).
          2 cpus - Intel(R) Pentium(R) 4 CPU 3.00GHz, no hyperthreading
          No write Cache, one core per cpu
          dual boot, with 28G disk space to linux.
          java: Java(TM) SE Runtime Environment (build pxi3260sr3-20081007_01(SR3))
          IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260-20081001_2388
          3 (JIT enabled, AOT enabled)
          J9VM - 20081001_023883_lHdSMr
          JIT - r9_20080930_1330
          GC - 20080926_AA)
          JCL - 20080924_01

          I've used the same nightly build: version: 10.5.0.0 alpha - (745360)

          My next step is to run on the linux box with just the
          derby.stream.error.logSeverityLevel=0
          derby.language.logStatementText=true
          in derby.properties, and on the linux box with the interrupts happening more often.

          For now, looks like we're no closer to figuring out what's going on with this.

          Show
          Myrna van Lunteren added a comment - Thank you, Kristian and Kathey, for running the test... My latest attempts have shown no unexpected errors except the GRANT. This setup different from the original in that I replace the derbyTesting.jar with a more recent one that also had the extra time printing (as per my patch) added derby.properties: derby.stream.error.logSeverityLevel=0 derby.language.logStatementText=true Apparently, one of these differences makes the problem disappear. I've also looked at the databases of the runs where there was some problem, and in 2 cases I had no problem selecting from any of the tables (user tables and system tables), and in the 2 cases where I got the XSDBB error once the corrupted table was REFRESH.INBOX and the other REFRESH.ATTACH. In each case the backed up database allowed select on all the tables. I've run now on 2 machines. The failing situations were on: windows machine: OS: Microsoft Windows 2000, service Pack 4 java: java version "1.6.0" Java(TM) SE Runtime Environment (build pwi3260sr2-20080607_01(SR2)) IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows 2000 x86-32 jvmwi3260-20080603_19964 (JIT enabled, AOT enabled) J9VM - 20080603_019964_lHdSMr JIT - r9_20080602_1330 GC - 20080603_AB) JCL - 20080606_02 IBM eserver, Two cpu's, x86 Family 15 Model 2 Stepping 7 GenuiniteIntel ~2790 Mhz, no hyperthreading No Write cache One core per cpu drives: 1 36gb scsi 1 72gbscsi (I'm running on the 62gb drive). Linux: Suse 9.1 (I know, oldish). 2 cpus - Intel(R) Pentium(R) 4 CPU 3.00GHz, no hyperthreading No write Cache, one core per cpu dual boot, with 28G disk space to linux. java: Java(TM) SE Runtime Environment (build pxi3260sr3-20081007_01(SR3)) IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260-20081001_2388 3 (JIT enabled, AOT enabled) J9VM - 20081001_023883_lHdSMr JIT - r9_20080930_1330 GC - 20080926_AA) JCL - 20080924_01 I've used the same nightly build: version: 10.5.0.0 alpha - (745360) My next step is to run on the linux box with just the derby.stream.error.logSeverityLevel=0 derby.language.logStatementText=true in derby.properties, and on the linux box with the interrupts happening more often. For now, looks like we're no closer to figuring out what's going on with this.
          Hide
          Myrna van Lunteren added a comment -

          Attaching the derby.log from the second run. I initially thought this one was less interesting then my first and third run failures (which caused the XSDBB error), because it was 'only' an 38000 error, but on second look this one has some thread dumps in it. Perhaps someone can deduce/suggest an avenue to reproduce the problem from that.

          Show
          Myrna van Lunteren added a comment - Attaching the derby.log from the second run. I initially thought this one was less interesting then my first and third run failures (which caused the XSDBB error), because it was 'only' an 38000 error, but on second look this one has some thread dumps in it. Perhaps someone can deduce/suggest an avenue to reproduce the problem from that.
          Hide
          Myrna van Lunteren added a comment -

          Attaching a jar with the output files and derby.log file of yet another run. This one is from windows, with modified test code, but without any derby.properties set.
          We got error:
          ERROR 38000: The exception 'java.sql.SQLException: Java exception: 'ASSERT FAILED Long column is not at the expected 0 slot, instead at slot -1: org.apache.derby.shared.common.sanity.AssertFailure'.' was thrown while evaluating an expression.

          This assert also printed out some thread stacks, and it seems that a refresh thread was running at the same time that compress was busy.

          Show
          Myrna van Lunteren added a comment - Attaching a jar with the output files and derby.log file of yet another run. This one is from windows, with modified test code, but without any derby.properties set. We got error: ERROR 38000: The exception 'java.sql.SQLException: Java exception: 'ASSERT FAILED Long column is not at the expected 0 slot, instead at slot -1: org.apache.derby.shared.common.sanity.AssertFailure'.' was thrown while evaluating an expression. This assert also printed out some thread stacks, and it seems that a refresh thread was running at the same time that compress was busy.
          Hide
          Myrna van Lunteren added a comment -

          Another Linux run ran into a NullPointer exception - but that was after running out of disk space. So I don't think that info is helpful. Looking at the errors so far, it seems likely the first Linux failure was also something else.

          Show
          Myrna van Lunteren added a comment - Another Linux run ran into a NullPointer exception - but that was after running out of disk space. So I don't think that info is helpful. Looking at the errors so far, it seems likely the first Linux failure was also something else.
          Hide
          Kristian Waagan added a comment -

          Myrna,

          o How frequently do you see the error(s)? Is it on every run, or more like one out of every ten?
          o If you see them frequently, can you spare a few cycles to run with the Sun VM as well?

          It's hard to tell, but since you can reproduce this on two different machines and operating systems, I'm guessing the bug is either VM or timing dependent.
          The problem is that by changing VM, you are most likely affecting the timing too...

          The only thing I have to offer at this point, is to keep running and restarting the test after about 24 hours. It is correct that you have seen the error within a day, right?
          If you share your latest findings, I'll configure my environment in a similar way and start the tests again.

          Show
          Kristian Waagan added a comment - Myrna, o How frequently do you see the error(s)? Is it on every run, or more like one out of every ten? o If you see them frequently, can you spare a few cycles to run with the Sun VM as well? It's hard to tell, but since you can reproduce this on two different machines and operating systems, I'm guessing the bug is either VM or timing dependent. The problem is that by changing VM, you are most likely affecting the timing too... The only thing I have to offer at this point, is to keep running and restarting the test after about 24 hours. It is correct that you have seen the error within a day, right? If you share your latest findings, I'll configure my environment in a similar way and start the tests again.
          Hide
          Myrna van Lunteren added a comment - - edited

          I think I've run this test in its non-intended configuration (i.e. none of the properties as specified in the checked-in derby.properties) about 7 times with sane build and had failures, 5 on windows, 2 on linux. Only 2x did I see the XSDBB errors.

          Findings so far:

          • on my windows box, error shows up after 24-27 hours
            (which effectively usually means I let it run for close to 2 days).
          • on windows: 5x seen some corruption error
          • on linux: 1 failure showed up after 15 hours.
          • on linux: 1 failure just before disk space full. Still investigating this.
          • any runs with logSeverityLevel=0/logStatementText were fine (ran for 3, 4 days before I killed it or it filled up the about 30G disk space I have available)
          • runs with modified derbyTesting.jar (print timing as per patch): 1x no problems, 1 time some corruption error.
          • insane build showed no errors.
          • linux vs. windows error situation is quite different - may not be the same problem.

          I'll do some more analysis of failing databases and log files, and kick off another test once I've run. And yes, I'll try with another jvm version. Note that I've run with different SR levels of the jvm on windows vs. linux. But I'll try a run with ibm15 and one with jdk16.

          Show
          Myrna van Lunteren added a comment - - edited I think I've run this test in its non-intended configuration (i.e. none of the properties as specified in the checked-in derby.properties) about 7 times with sane build and had failures, 5 on windows, 2 on linux. Only 2x did I see the XSDBB errors. Findings so far: on my windows box, error shows up after 24-27 hours (which effectively usually means I let it run for close to 2 days). on windows: 5x seen some corruption error on linux: 1 failure showed up after 15 hours. on linux: 1 failure just before disk space full. Still investigating this. any runs with logSeverityLevel=0/logStatementText were fine (ran for 3, 4 days before I killed it or it filled up the about 30G disk space I have available) runs with modified derbyTesting.jar (print timing as per patch): 1x no problems, 1 time some corruption error. insane build showed no errors. linux vs. windows error situation is quite different - may not be the same problem. I'll do some more analysis of failing databases and log files, and kick off another test once I've run. And yes, I'll try with another jvm version. Note that I've run with different SR levels of the jvm on windows vs. linux. But I'll try a run with ibm15 and one with jdk16.
          Hide
          Knut Anders Hatlen added a comment -

          I've had the test running for a day with the sleep call mentioned in DERBY-3393 added to see if that could help us reproducing it. Haven't seen any errors so far, so that's probably not a related issue.

          Show
          Knut Anders Hatlen added a comment - I've had the test running for a day with the sleep call mentioned in DERBY-3393 added to see if that could help us reproducing it. Haven't seen any errors so far, so that's probably not a related issue.
          Hide
          Kathey Marsden added a comment -

          Well I ran the test on a 4 CPU Windows 200 box and ran out of space after 3 days, but no corruption. I had 30GB available, but I guess that wasn't enough, especially with the test trying to make a backup of the database. Is it possible this test is hitting DERBY-4055? The jvm I used was:

          java version "1.6.0"
          Java(TM) SE Runtime Environment (build pwi3260sr3-20081106_07(SR3))
          IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows 2000 x86-32 jvmwi3260-200811
          05_25433 (JIT enabled, AOT enabled)
          J9VM - 20081105_025433_lHdSMr
          JIT - r9_20081031_1330
          GC - 20081027_AB)
          JCL - 20081106_01

          I noticed the jvm version against which this was reported build pwi3260sr2-20080607_01(SR2)) is quite old. I think we should probably discount these results. I noticed IBM now has an SR4 available too. It would be interesting to see if we could reproduce the issue on the machine where we saw the problem with SR4.

          Show
          Kathey Marsden added a comment - Well I ran the test on a 4 CPU Windows 200 box and ran out of space after 3 days, but no corruption. I had 30GB available, but I guess that wasn't enough, especially with the test trying to make a backup of the database. Is it possible this test is hitting DERBY-4055 ? The jvm I used was: java version "1.6.0" Java(TM) SE Runtime Environment (build pwi3260sr3-20081106_07(SR3)) IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows 2000 x86-32 jvmwi3260-200811 05_25433 (JIT enabled, AOT enabled) J9VM - 20081105_025433_lHdSMr JIT - r9_20081031_1330 GC - 20081027_AB) JCL - 20081106_01 I noticed the jvm version against which this was reported build pwi3260sr2-20080607_01(SR2)) is quite old. I think we should probably discount these results. I noticed IBM now has an SR4 available too. It would be interesting to see if we could reproduce the issue on the machine where we saw the problem with SR4.
          Hide
          Mike Matrigali added a comment -

          I ran for about 3.5 days on a 4 cpu windows xp box and ran of space also. But
          before that got no corruption errors. The db had grown to around 17gb as
          reported by the test. Note for those running the test, it does not catch
          some errors and stop. In my case it continued to run reporting no connection
          for every action after the backup failed for out of disk space.

          I am running against sane jars from the release candidate. And configured
          the test "incorrectly" as myrna described in her initial reports - ie. no
          property file included and am running embedded using the following:

          java org.apache.derbyTesting.system.mailjdbc.MailJdbc embedded > scenario.out 2>
          &1

          The jvm I used was:
          java version "1.6.0"
          Java(TM) SE Runtime Environment (build pwi3260sr3-20081106_07(SR3))
          IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260-20081105
          _25433 (JIT enabled, AOT enabled)
          J9VM - 20081105_025433_lHdSMr
          JIT - r9_20081031_1330
          GC - 20081027_AB)
          JCL - 20081106_01

          I also think we should discount the old jvm results and lower the priority of
          this issue unless someone can reproduce against a current jvm.

          I started another run, today.

          Has anyone run this test for awhile in sane and configured as above against a
          release before 10.5. I am wondering if the growing db is a new behavior or
          not.

          Show
          Mike Matrigali added a comment - I ran for about 3.5 days on a 4 cpu windows xp box and ran of space also. But before that got no corruption errors. The db had grown to around 17gb as reported by the test. Note for those running the test, it does not catch some errors and stop. In my case it continued to run reporting no connection for every action after the backup failed for out of disk space. I am running against sane jars from the release candidate. And configured the test "incorrectly" as myrna described in her initial reports - ie. no property file included and am running embedded using the following: java org.apache.derbyTesting.system.mailjdbc.MailJdbc embedded > scenario.out 2> &1 The jvm I used was: java version "1.6.0" Java(TM) SE Runtime Environment (build pwi3260sr3-20081106_07(SR3)) IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260-20081105 _25433 (JIT enabled, AOT enabled) J9VM - 20081105_025433_lHdSMr JIT - r9_20081031_1330 GC - 20081027_AB) JCL - 20081106_01 I also think we should discount the old jvm results and lower the priority of this issue unless someone can reproduce against a current jvm. I started another run, today. Has anyone run this test for awhile in sane and configured as above against a release before 10.5. I am wondering if the growing db is a new behavior or not.
          Hide
          Myrna van Lunteren added a comment - - edited

          I ran on my linux machine with ibm15 and there was no problem except that the test ran out of disk space.
          I am running on the windows machine where I saw the original problem with Sun's 1.6 jvm (an old version, but shouldn't matter) and so far, no problem (needs a couple more hours).
          I am lowering the priority; it seems to me enough people have run this test to indicate that it's certainly no common problem; that it doesn't happen with insane builds; and that it's possibly a (old) jvm issue.
          I'll do a few more runs with different jvms/jvm versions.

          Show
          Myrna van Lunteren added a comment - - edited I ran on my linux machine with ibm15 and there was no problem except that the test ran out of disk space. I am running on the windows machine where I saw the original problem with Sun's 1.6 jvm (an old version, but shouldn't matter) and so far, no problem (needs a couple more hours). I am lowering the priority; it seems to me enough people have run this test to indicate that it's certainly no common problem; that it doesn't happen with insane builds; and that it's possibly a (old) jvm issue. I'll do a few more runs with different jvms/jvm versions.
          Hide
          Kathey Marsden added a comment -

          I will kick off a run with the 10.4.2.0 debug/sane jars to see if the growth issue was preexisting.

          Show
          Kathey Marsden added a comment - I will kick off a run with the 10.4.2.0 debug/sane jars to see if the growth issue was preexisting.
          Hide
          Myrna van Lunteren added a comment -

          To my disappointment my jdk16 (rc0, yes, I know, old) sane run on the windows machine where I first saw the problem, also showed an XSDBB error.
          Attaching the whole shabang of the test dir.
          After the XSDBB error, we get some errors worrying that there's no more disk space - but there was 25G left when I checked, I don't think that's it.

          Running with ibm16 sr3 now.

          Show
          Myrna van Lunteren added a comment - To my disappointment my jdk16 (rc0, yes, I know, old) sane run on the windows machine where I first saw the problem, also showed an XSDBB error. Attaching the whole shabang of the test dir. After the XSDBB error, we get some errors worrying that there's no more disk space - but there was 25G left when I checked, I don't think that's it. Running with ibm16 sr3 now.
          Hide
          Myrna van Lunteren added a comment -

          I saw the linux fail after (almost) 2 days with ibm 1.6 sr4.
          The ibm16 sr3 windows run is still good after 2 days...

          Show
          Myrna van Lunteren added a comment - I saw the linux fail after (almost) 2 days with ibm 1.6 sr4. The ibm16 sr3 windows run is still good after 2 days...
          Hide
          Myrna van Lunteren added a comment -

          attaching full zipped file of first windows failure (ibm sr2)

          Show
          Myrna van Lunteren added a comment - attaching full zipped file of first windows failure (ibm sr2)
          Hide
          Myrna van Lunteren added a comment -

          attaching zipped dbs & log & output files from third failed windows run.

          Show
          Myrna van Lunteren added a comment - attaching zipped dbs & log & output files from third failed windows run.
          Hide
          Myrna van Lunteren added a comment -

          attaching full dir of liirst linux failure

          Show
          Myrna van Lunteren added a comment - attaching full dir of liirst linux failure
          Hide
          Myrna van Lunteren added a comment -

          I wanted to upload 3 more windows failure test runs, but I've exceeded the maximum for disk space for all files combined for this issue.
          If anyone is interested in more output files I can put some more zip files in my home directory at apache....

          Show
          Myrna van Lunteren added a comment - I wanted to upload 3 more windows failure test runs, but I've exceeded the maximum for disk space for all files combined for this issue. If anyone is interested in more output files I can put some more zip files in my home directory at apache....
          Hide
          Myrna van Lunteren added a comment -

          My original reports were with 10.5: 745360 on a machine running windows. My latest run, with the 10.5 RC2 candidate (10.5.1.1.764942) on the windows machine did not show any corruption until the disk ran out of space. So - perhaps that was a bad build after all?

          A similar run on the linux machine did show corruption, however, so I'm not closing this yet.

          Show
          Myrna van Lunteren added a comment - My original reports were with 10.5: 745360 on a machine running windows. My latest run, with the 10.5 RC2 candidate (10.5.1.1.764942) on the windows machine did not show any corruption until the disk ran out of space. So - perhaps that was a bad build after all? A similar run on the linux machine did show corruption, however, so I'm not closing this yet.
          Hide
          Myrna van Lunteren added a comment -

          I ran with sane builds of 10.6.0.0 alpha revision 781651 on the two machines where I initially saw this problem, same jvm, same environment, and did not see the corruption!
          I am marking this as a Duplicate for DERBY-4239.

          Show
          Myrna van Lunteren added a comment - I ran with sane builds of 10.6.0.0 alpha revision 781651 on the two machines where I initially saw this problem, same jvm, same environment, and did not see the corruption! I am marking this as a Duplicate for DERBY-4239 .
          Hide
          Myrna van Lunteren added a comment -

          Note that on my Windows I saw a number of deadlock warnings while I didn't see them on my Linux machine. But that's outside the scope of this issue.

          Show
          Myrna van Lunteren added a comment - Note that on my Windows I saw a number of deadlock warnings while I didn't see them on my Linux machine. But that's outside the scope of this issue.
          Hide
          Kristian Waagan added a comment -

          Added link to the duplicate.

          Show
          Kristian Waagan added a comment - Added link to the duplicate.

            People

            • Assignee:
              Unassigned
              Reporter:
              Myrna van Lunteren
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development