Derby
  1. Derby
  2. DERBY-5234

Unable to insert data into table. Failed due be "ERROR XSDG0: Page Page(51919,Container(0, 1104)) could not be read from disk."

    Details

    • Urgency:
      Urgent
    • Bug behavior facts:
      Data corruption

      Description

      One of the derby database table "gets corrupted"/"indicates connection not available" during processing inserts from java client application as shown in the trace and the only way to recover from this error is to rebuild the DB - by deleting the data and creating the tables again. This happens once in a while (thrice in a span of two months) and the java application (run in multiple servers), which updates the database, processes around 100 million transactions per hour (in total and each transation results in 4-5 updates to the DB)

      There are eight tables in the derby database.

      TABLE NAME ROWS COUNT (at time of corruption)
      ---------------------------------------------------------------------------------
      KPI.KPI_MERGEIN; 362917
      KPI.KPI_IN; 422508
      KPI.KPI_DROPPED; 53667
      KPI.KPI_ERROR1; 0
      KPI.KPI_ERROR2; 2686
      KPI.KPI_ERRORMERGE; 0
      KPI.KPI_MERGEOUT; 362669
      KPI.KPI_OUT; 125873

      The derby database has been started with the following parameters

      CMD="java -Dderby.system.home=$DERBY_OPTS -Dderby.locks.monitor=true -Dderby.locks.deadlockTrace=true -Dderby.locks.escalationThreshold=50000 -Dderby.locks.waitTimeout=
      -1 -Dderby.storage.pageCacheSize=100000 -Xms512M -Xmx3072M -XX:NewSize=256M -classpath $DERBY_CLASSPATH org.apache.derby.drda.NetworkServerControl start -h $KPIDERBYHOST -p $DERBY_KPI_PORT"

      The corrupted database tar (filesystem) in live environment was moved to a test system (Solaris system) and few checks were run on the corrupted DB as part of analysis (DB does start fine)

      While trying to insert a row in any table expect KPI.KPI_MERGEIN, it is successful. But when a new row is inserted into KPI.KPI_MERGEIN table using command line tool it's throwing below error message (the same message that appeared in live

      ij> INSERT INTO KPI.KPI_MERGEIN (A0_TXN_ID, A1_NE_ID, A2_CHU_IP_ADDR, A3_BATCH_DATE,A5_CODE) VALUES (-1, 'BMTDE', '192.2.1.3', 231456879, 'KSD');
      ERROR 08006: A network protocol error was encountered and the connection has been terminated: the requested command encountered an unarchitected and implementation-specific condition for which there was no architected message

      and in derby.log file it shows below error stacktrace.

      ERROR XSDG0: Page Page(51919,Container(0, 1104)) could not be read from disk.
      at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(Unknown Source)
      at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(Unknown Source)
      at org.apache.derby.impl.services.cache.ConcurrentCache.find(Unknown Source)
      at org.apache.derby.impl.store.raw.data.FileContainer.initPage(Unknown Source)
      at org.apache.derby.impl.store.raw.data.FileContainer.newPage(Unknown Source)
      at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(Unknown Source)
      at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(Unknown Source)
      at org.apache.derby.impl.store.access.heap.HeapController.doInsert(Unknown Source)
      at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(Unknown Source)
      at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(Unknown Source)
      at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(Unknown Source)
      at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source)
      at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
      at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLIMM(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
      Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(Unknown Source)
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(Unknown Source)
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown Source)
      ... 20 more
      ============= begin nested exception, level (1) ===========
      java.io.EOFException: Reached end of file while attempting to read a whole page.
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(Unknown Source)
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(Unknown Source)
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown Source)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(Unknown Source)
      at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(Unknown Source)
      at org.apache.derby.impl.services.cache.ConcurrentCache.find(Unknown Source)
      at org.apache.derby.impl.store.raw.data.FileContainer.initPage(Unknown Source)
      at org.apache.derby.impl.store.raw.data.FileContainer.newPage(Unknown Source)
      at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(Unknown Source)
      at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(Unknown Source)
      at org.apache.derby.impl.store.access.heap.HeapController.doInsert(Unknown Source)
      at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(Unknown Source)
      at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(Unknown Source)
      at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(Unknown Source)
      at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source)
      at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
      at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLIMM(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
      ============= end nested exception, level (1) ===========

      2011-05-16 10:37:21.392 GMT:
      Shutting down instance a816c00e-012f-f85f-7892-ffff874c3ff6
      ----------------------------------------------------------------
      Cleanup action completed

      The problem is only with INSERT statement. When i try SELECT statement on KPI.KPI_MERGEIN table it is working well.The database file system size (in seg0) is 1.3 GB

      Can anyone help me out in identifying the problem that why for one table alone its throwing the above error message ? Would upgrade to a new version help ?

      1. log85.dat
        1.00 MB
        Rick Hillegas
      2. log191.dat
        1024 kB
        Rick Hillegas
      3. derby-5234-02-aa-edgeCaseTests.diff
        4 kB
        Rick Hillegas
      4. derby-5234-01-ab-emptyAllocPage.diff
        9 kB
        Rick Hillegas
      5. derby-5234-01-aa-emptyAllocPage.diff
        2 kB
        Rick Hillegas
      6. Derby5234.java
        7 kB
        Rick Hillegas
      7. Derby5234.java
        10 kB
        Rick Hillegas
      8. DbCompressErrorTester.java
        6 kB
        Stefan Sitte
      9. DataFileReader_Output.zip
        840 kB
        Varma R
      10. 5234_summary.out
        57 kB
        Rick Hillegas
      11. 5234_page_10219.out
        310 kB
        Rick Hillegas
      12. 5234_alloc.out
        8.20 MB
        Rick Hillegas

        Issue Links

          Activity

          Hide
          ASF subversion and git services added a comment -

          Commit 1533682 from Kathey Marsden in branch 'code/branches/10.5'
          [ https://svn.apache.org/r1533682 ]

          DERBY-6382 fter Inplace compress: java.sql.SQLException: Page Page(10219,Container(0, 1136)) could not be read from disk caused by Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.

          This is a partial fix for DERBY-5234. Merged revision 1337258 from 10.8
          Contributed by Rick Hillegas

          Show
          ASF subversion and git services added a comment - Commit 1533682 from Kathey Marsden in branch 'code/branches/10.5' [ https://svn.apache.org/r1533682 ] DERBY-6382 fter Inplace compress: java.sql.SQLException: Page Page(10219,Container(0, 1136)) could not be read from disk caused by Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page. This is a partial fix for DERBY-5234 . Merged revision 1337258 from 10.8 Contributed by Rick Hillegas
          Hide
          ASF subversion and git services added a comment -

          Commit 1533540 from Kathey Marsden in branch 'code/branches/10.6'
          [ https://svn.apache.org/r1533540 ]

          DERBY-6382 fter Inplace compress: java.sql.SQLException: Page Page(10219,Container(0, 1136)) could not be read from disk caused by Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.

          This is a partial fix for DERBY-5234. Merged revision 1337258 from 10.8
          Contributed by Rick Hillegas

          Show
          ASF subversion and git services added a comment - Commit 1533540 from Kathey Marsden in branch 'code/branches/10.6' [ https://svn.apache.org/r1533540 ] DERBY-6382 fter Inplace compress: java.sql.SQLException: Page Page(10219,Container(0, 1136)) could not be read from disk caused by Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page. This is a partial fix for DERBY-5234 . Merged revision 1337258 from 10.8 Contributed by Rick Hillegas
          Hide
          ASF subversion and git services added a comment -

          Commit 1533412 from Kathey Marsden in branch 'code/branches/10.7'
          [ https://svn.apache.org/r1533412 ]

          DERBY-6382 fter Inplace compress: java.sql.SQLException: Page Page(10219,Container(0, 1136)) could not be read from disk caused by Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.

          This is a partial fix for DERBY-5234. Merged revision 1337258 from 10.8
          Contributed by Rick Hillegas

          Show
          ASF subversion and git services added a comment - Commit 1533412 from Kathey Marsden in branch 'code/branches/10.7' [ https://svn.apache.org/r1533412 ] DERBY-6382 fter Inplace compress: java.sql.SQLException: Page Page(10219,Container(0, 1136)) could not be read from disk caused by Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page. This is a partial fix for DERBY-5234 . Merged revision 1337258 from 10.8 Contributed by Rick Hillegas
          Hide
          Kathey Marsden added a comment -

          So, although it was closed CannotReproduce, I think something was fixed in this issue. 1335570 and 1335677 were committed to trunk and ported to 10.8 branch at subversion revision 1337258. For bookkeeping and possible further backport it would seem to make sense to open a separate issue for what was fixed.

          Show
          Kathey Marsden added a comment - So, although it was closed CannotReproduce, I think something was fixed in this issue. 1335570 and 1335677 were committed to trunk and ported to 10.8 branch at subversion revision 1337258. For bookkeeping and possible further backport it would seem to make sense to open a separate issue for what was fixed.
          Hide
          Rick Hillegas added a comment -

          Knut's understanding of the bug agrees with mine. There seems to be consensus that the table compression code gives rise to many data corruptions and that code needs to be overhauled. That's why this issue was linked to DERBY-5876, an issue which describes a systemic rather than piecemeal mini-project for re-writing that part of Derby. Thanks.

          Show
          Rick Hillegas added a comment - Knut's understanding of the bug agrees with mine. There seems to be consensus that the table compression code gives rise to many data corruptions and that code needs to be overhauled. That's why this issue was linked to DERBY-5876 , an issue which describes a systemic rather than piecemeal mini-project for re-writing that part of Derby. Thanks.
          Hide
          Knut Anders Hatlen added a comment -

          The state of this bug report is a bit confusing, since it contains multiple reports of issues with similar symptoms. Those issues may or may not be related. One fix has been checked in and seems to have fixed one of the issues, but it also sounds like at least one of the problems remains unsolved.

          My reading of the comments on this bug report is that the issue Stefan saw could be reproduced, and it has been fixed. That fix is included in the 10.8.3.0 maintenance release. However, the problems reported by Varma and Saurabh have not been reproduced by anyone else yet. Saurabh has tested the fix and concluded that the fix for Stefan's issue did not fix the issue he saw. It is not known whether or not the fix addresses the problem reported by Varma.

          Does that accurately describe the current state of the bug report?

          Show
          Knut Anders Hatlen added a comment - The state of this bug report is a bit confusing, since it contains multiple reports of issues with similar symptoms. Those issues may or may not be related. One fix has been checked in and seems to have fixed one of the issues, but it also sounds like at least one of the problems remains unsolved. My reading of the comments on this bug report is that the issue Stefan saw could be reproduced, and it has been fixed. That fix is included in the 10.8.3.0 maintenance release. However, the problems reported by Varma and Saurabh have not been reproduced by anyone else yet. Saurabh has tested the fix and concluded that the fix for Stefan's issue did not fix the issue he saw. It is not known whether or not the fix addresses the problem reported by Varma. Does that accurately describe the current state of the bug report?
          Hide
          Stefan Sitte added a comment -

          I am very disappointed that even after one year you know this error and have a repro you closed the ticket.
          I'm wondering why the problems regarding the compression (inplace and normal) won't be fixed. In my opinion this is essential for running a database.

          Show
          Stefan Sitte added a comment - I am very disappointed that even after one year you know this error and have a repro you closed the ticket. I'm wondering why the problems regarding the compression (inplace and normal) won't be fixed. In my opinion this is essential for running a database.
          Hide
          Rick Hillegas added a comment -

          Without more information, we can't reproduce this problem. We can re-open this issue if more information becomes available.

          Show
          Rick Hillegas added a comment - Without more information, we can't reproduce this problem. We can re-open this issue if more information becomes available.
          Hide
          Rick Hillegas added a comment -

          Linking this issue to DERBY-5876, a master issue for overhauling the table compression code. Hopefully the underlying issue with this bug will be addressed when we revamp table compression. However, without a repro we won't be able to prove that. Now that this issue is linked to DERBY-5876, I don't see much point in keeping this unreproducible bug open. Thanks.

          Show
          Rick Hillegas added a comment - Linking this issue to DERBY-5876 , a master issue for overhauling the table compression code. Hopefully the underlying issue with this bug will be addressed when we revamp table compression. However, without a repro we won't be able to prove that. Now that this issue is linked to DERBY-5876 , I don't see much point in keeping this unreproducible bug open. Thanks.
          Hide
          Kathey Marsden added a comment -

          Turning off repro attached and high value fix as from the comments it seems the attempted reproductions attached did not reproduce the issue for Rick. The user resorted to working around the problem. It would be great to get a reproduction for the issue so we could fix it, but until then there is not much we can do.
          Should we close this cannot reproduce?

          Show
          Kathey Marsden added a comment - Turning off repro attached and high value fix as from the comments it seems the attempted reproductions attached did not reproduce the issue for Rick. The user resorted to working around the problem. It would be great to get a reproduction for the issue so we could fix it, but until then there is not much we can do. Should we close this cannot reproduce?
          Hide
          Kathey Marsden added a comment -

          Removing fix versions as from the comments it doesn't seem to be fixed.

          Show
          Kathey Marsden added a comment - Removing fix versions as from the comments it doesn't seem to be fixed.
          Hide
          Rick Hillegas added a comment -

          Unassigning myself from this issue because I obviously haven't done anything on this issue for a long time. My sense is that the table compression code has a number of weaknesses and we should devote some serious effort to overhauling it.

          Show
          Rick Hillegas added a comment - Unassigning myself from this issue because I obviously haven't done anything on this issue for a long time. My sense is that the table compression code has a number of weaknesses and we should devote some serious effort to overhauling it.
          Hide
          Saurabh Singhi added a comment -

          I have given up as well. Resorting to dynamic creation and deletion of tables (as opposed to rows) for now.

          Show
          Saurabh Singhi added a comment - I have given up as well. Resorting to dynamic creation and deletion of tables (as opposed to rows) for now.
          Hide
          Rick Hillegas added a comment -

          Thanks for the deadlock trace Saurabh. It looks similar to the stack trace for DERBY-3683. However, the explicit table lock doesn't help in your case. I am linking this issue to the other issues which prevent you from using SYSCS_COMPRESS_TABLE: DERBY-3683 and DERBY-5358. Until we make progress on those issues, I am unable to recommend a workaround. Thanks.

          Show
          Rick Hillegas added a comment - Thanks for the deadlock trace Saurabh. It looks similar to the stack trace for DERBY-3683 . However, the explicit table lock doesn't help in your case. I am linking this issue to the other issues which prevent you from using SYSCS_COMPRESS_TABLE: DERBY-3683 and DERBY-5358 . Until we make progress on those issues, I am unable to recommend a workaround. Thanks.
          Hide
          Saurabh Singhi added a comment -

          Hi Rick,
          My program created a new 10.8.2.3 database along with the regular select, delete, lock, compress and insert statements.
          As per your suggestions, the program executed a LOCK statement (EXCLUSIVE lock) before executing the SYSCS_COMPRESS_TABLE call.
          I'm still getting the deadlock exception (while selecting and inserting) shown below:-

          ***********************************************************************************************************************************************************
          22 May 2012 13:37:32 ProcessDAO.getLastRecordFromIntervalTableShowGap = A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
          Lock : TABLE, PROCESS_055234_HR, Tablelock
          Waiting XID :

          {104215, IS} , APP, SELECT CpuNum, SampleTimestamp AS TimeStamp, ProcBusyPercent AS BusyPercent FROM PROCESS_055234_HR WHERE SampleTimestamp >= ? AND Volume=? AND Subvol=? AND Filename=? AND ProcessName=? AND CpuNum=? AND PIN=? ORDER BY SampleTimestamp
          Granted XID : {102377, X}
          Lock : ROW, SYSCONGLOMERATES, (7,9)
          Waiting XID : {102377, X} , APP, alter table "APP"."PROCESS_055234_HR" compress sequential
          Granted XID : {102377, S} , {104215, S}
          . The selected victim is XID : 104215.
          java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
          Lock : TABLE, PROCESS_055234_HR, Tablelock
          Waiting XID : {104215, IS}

          , APP, SELECT CpuNum, SampleTimestamp AS TimeStamp, ProcBusyPercent AS BusyPercent FROM PROCESS_055234_HR WHERE SampleTimestamp >= ? AND Volume=? AND Subvol=? AND Filename=? AND ProcessName=? AND CpuNum=? AND PIN=? ORDER BY SampleTimestamp
          Granted XID :

          {102377, X}
          Lock : ROW, SYSCONGLOMERATES, (7,9)
          Waiting XID : {102377, X}

          , APP, alter table "APP"."PROCESS_055234_HR" compress sequential
          Granted XID :

          {102377, S} , {104215, S}
          . The selected victim is XID : 104215.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:89)
          at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Util.java:256)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:424)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290)
          at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:82)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(EmbedPreparedStatement.java:151)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(EmbedPreparedStatement20.java:82)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(EmbedPreparedStatement30.java:63)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.<init>(EmbedPreparedStatement40.java:40)
          at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Driver40.java:107)
          at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(EmbedConnection.java:1615)
          at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(EmbedConnection.java:1443)
          at com.idelji.operations.iasset.plugin.imon.dao.ProcessDAO.getLastRecordFromIntervalTableShowGap(ProcessDAO.java:503)
          at imon.utils.DataSourceRouter.getLatestProcessData(DataSourceRouter.java:608)
          at imon.db.ProcessDataProvider.getLatestData(ProcessDataProvider.java:131)
          at imon.charts.LineChartWithSlider.refreshChartData(LineChartWithSlider.java:558)
          at imon.charts.LineChartWithSlider$1.handle(LineChartWithSlider.java:327)
          at imon.charts.LineChartWithSlider$1.handle(LineChartWithSlider.java:322)
          at com.sun.scenario.animation.shared.TimelineClipCore.visitKeyFrame(TimelineClipCore.java:196)
          at com.sun.scenario.animation.shared.TimelineClipCore.playTo(TimelineClipCore.java:137)
          at javafx.animation.Timeline.impl_playTo(Timeline.java:163)
          at com.sun.scenario.animation.shared.InfiniteClipEnvelope.timePulse(InfiniteClipEnvelope.java:89)
          at javafx.animation.Animation.impl_timePulse(Animation.java:952)
          at com.sun.scenario.animation.shared.AnimationPulseReceiver.timePulse(AnimationPulseReceiver.java:75)
          at com.sun.scenario.animation.AbstractMasterTimer.timePulseImpl(AbstractMasterTimer.java:357)
          at com.sun.scenario.animation.AbstractMasterTimer$MainLoop.run(AbstractMasterTimer.java:280)
          at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(QuantumToolkit.java:381)
          at com.sun.javafx.tk.quantum.QuantumToolkit$8.run(QuantumToolkit.java:311)
          at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
          at com.sun.glass.ui.win.WinApplication.access$100(WinApplication.java:29)
          at com.sun.glass.ui.win.WinApplication$2$1.run(WinApplication.java:62)
          at java.lang.Thread.run(Unknown Source)

          22 May 2012 13:37:52 ProcessDAO.insertData = A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
          Lock : ROW, SYSCONGLOMERATES, (7,9)
          Waiting XID : {104576, S} , APP, INSERT INTO PROCESS_055234_HR (SampleTimestamp,CpuNum,Pin,Priority,UserID,GroupID,ProcessName,Volume,SubVol,FileName,AncestorProcessName,ProcBusyPercent) VALUES (?,?,?,?,?,?,?,?,?,?,?,?)
          Granted XID : {102377, S}


          Lock : ROW, SYSCONGLOMERATES, (7,8)
          Waiting XID :

          {102377, X} , APP, alter table "APP"."PROCESS_055234_HR" compress sequential
          Granted XID : {102377, S} , {104576, S}
          . The selected victim is XID : 104576.
          java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
          Lock : ROW, SYSCONGLOMERATES, (7,9)
          Waiting XID : {104576, S} , APP, INSERT INTO PROCESS_055234_HR (SampleTimestamp,CpuNum,Pin,Priority,UserID,GroupID,ProcessName,Volume,SubVol,FileName,AncestorProcessName,ProcBusyPercent) VALUES (?,?,?,?,?,?,?,?,?,?,?,?)
          Granted XID : {102377, S}
          Lock : ROW, SYSCONGLOMERATES, (7,8)
          Waiting XID : {102377, X}

          , APP, alter table "APP"."PROCESS_055234_HR" compress sequential
          Granted XID :

          {102377, S}

          ,

          {104576, S}


          . The selected victim is XID : 104576.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:89)
          at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Util.java:256)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:424)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290)
          at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:82)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(EmbedPreparedStatement.java:151)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(EmbedPreparedStatement20.java:82)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(EmbedPreparedStatement30.java:63)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.<init>(EmbedPreparedStatement40.java:40)
          at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Driver40.java:107)
          at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(EmbedConnection.java:1615)
          at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(EmbedConnection.java:1443)
          at com.idelji.operations.iasset.plugin.imon.dao.ProcessDAO.insertData(ProcessDAO.java:86)
          at com.idelji.operations.iasset.plugin.imon.comm.CPUProcessDataStoreWriter.onData(CPUProcessDataStoreWriter.java:56)
          at com.idelji.operations.iasset.plugin.imon.handler.CPUProcessDataHandler.gotData(CPUProcessDataHandler.java:205)
          at com.idelji.operations.ibase.core.comm.Connector.makeRequest(Connector.java:83)
          at com.idelji.operations.ibase.core.comm.EntityConnector.makeRequest(EntityConnector.java:36)
          at com.idelji.operations.iasset.plugin.imon.comm.CPUProcessCommunicator.run(CPUProcessCommunicator.java:98)

          ***********************************************************************************************************************************************************

          Show
          Saurabh Singhi added a comment - Hi Rick, My program created a new 10.8.2.3 database along with the regular select, delete, lock, compress and insert statements. As per your suggestions, the program executed a LOCK statement (EXCLUSIVE lock) before executing the SYSCS_COMPRESS_TABLE call. I'm still getting the deadlock exception (while selecting and inserting) shown below:- *********************************************************************************************************************************************************** 22 May 2012 13:37:32 ProcessDAO.getLastRecordFromIntervalTableShowGap = A lock could not be obtained due to a deadlock, cycle of locks and waiters is: Lock : TABLE, PROCESS_055234_HR, Tablelock Waiting XID : {104215, IS} , APP, SELECT CpuNum, SampleTimestamp AS TimeStamp, ProcBusyPercent AS BusyPercent FROM PROCESS_055234_HR WHERE SampleTimestamp >= ? AND Volume=? AND Subvol=? AND Filename=? AND ProcessName=? AND CpuNum=? AND PIN=? ORDER BY SampleTimestamp Granted XID : {102377, X} Lock : ROW, SYSCONGLOMERATES, (7,9) Waiting XID : {102377, X} , APP, alter table "APP"."PROCESS_055234_HR" compress sequential Granted XID : {102377, S} , {104215, S} . The selected victim is XID : 104215. java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is: Lock : TABLE, PROCESS_055234_HR, Tablelock Waiting XID : {104215, IS} , APP, SELECT CpuNum, SampleTimestamp AS TimeStamp, ProcBusyPercent AS BusyPercent FROM PROCESS_055234_HR WHERE SampleTimestamp >= ? AND Volume=? AND Subvol=? AND Filename=? AND ProcessName=? AND CpuNum=? AND PIN=? ORDER BY SampleTimestamp Granted XID : {102377, X} Lock : ROW, SYSCONGLOMERATES, (7,9) Waiting XID : {102377, X} , APP, alter table "APP"."PROCESS_055234_HR" compress sequential Granted XID : {102377, S} , {104215, S} . The selected victim is XID : 104215. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:89) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Util.java:256) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:424) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:82) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(EmbedPreparedStatement.java:151) at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(EmbedPreparedStatement20.java:82) at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(EmbedPreparedStatement30.java:63) at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.<init>(EmbedPreparedStatement40.java:40) at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Driver40.java:107) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(EmbedConnection.java:1615) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(EmbedConnection.java:1443) at com.idelji.operations.iasset.plugin.imon.dao.ProcessDAO.getLastRecordFromIntervalTableShowGap(ProcessDAO.java:503) at imon.utils.DataSourceRouter.getLatestProcessData(DataSourceRouter.java:608) at imon.db.ProcessDataProvider.getLatestData(ProcessDataProvider.java:131) at imon.charts.LineChartWithSlider.refreshChartData(LineChartWithSlider.java:558) at imon.charts.LineChartWithSlider$1.handle(LineChartWithSlider.java:327) at imon.charts.LineChartWithSlider$1.handle(LineChartWithSlider.java:322) at com.sun.scenario.animation.shared.TimelineClipCore.visitKeyFrame(TimelineClipCore.java:196) at com.sun.scenario.animation.shared.TimelineClipCore.playTo(TimelineClipCore.java:137) at javafx.animation.Timeline.impl_playTo(Timeline.java:163) at com.sun.scenario.animation.shared.InfiniteClipEnvelope.timePulse(InfiniteClipEnvelope.java:89) at javafx.animation.Animation.impl_timePulse(Animation.java:952) at com.sun.scenario.animation.shared.AnimationPulseReceiver.timePulse(AnimationPulseReceiver.java:75) at com.sun.scenario.animation.AbstractMasterTimer.timePulseImpl(AbstractMasterTimer.java:357) at com.sun.scenario.animation.AbstractMasterTimer$MainLoop.run(AbstractMasterTimer.java:280) at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(QuantumToolkit.java:381) at com.sun.javafx.tk.quantum.QuantumToolkit$8.run(QuantumToolkit.java:311) at com.sun.glass.ui.win.WinApplication._runLoop(Native Method) at com.sun.glass.ui.win.WinApplication.access$100(WinApplication.java:29) at com.sun.glass.ui.win.WinApplication$2$1.run(WinApplication.java:62) at java.lang.Thread.run(Unknown Source) 22 May 2012 13:37:52 ProcessDAO.insertData = A lock could not be obtained due to a deadlock, cycle of locks and waiters is: Lock : ROW, SYSCONGLOMERATES, (7,9) Waiting XID : {104576, S} , APP, INSERT INTO PROCESS_055234_HR (SampleTimestamp,CpuNum,Pin,Priority,UserID,GroupID,ProcessName,Volume,SubVol,FileName,AncestorProcessName,ProcBusyPercent) VALUES (?,?,?,?,?,?,?,?,?,?,?,?) Granted XID : {102377, S} Lock : ROW, SYSCONGLOMERATES, (7,8) Waiting XID : {102377, X} , APP, alter table "APP"."PROCESS_055234_HR" compress sequential Granted XID : {102377, S} , {104576, S} . The selected victim is XID : 104576. java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is: Lock : ROW, SYSCONGLOMERATES, (7,9) Waiting XID : {104576, S} , APP, INSERT INTO PROCESS_055234_HR (SampleTimestamp,CpuNum,Pin,Priority,UserID,GroupID,ProcessName,Volume,SubVol,FileName,AncestorProcessName,ProcBusyPercent) VALUES (?,?,?,?,?,?,?,?,?,?,?,?) Granted XID : {102377, S} Lock : ROW, SYSCONGLOMERATES, (7,8) Waiting XID : {102377, X} , APP, alter table "APP"."PROCESS_055234_HR" compress sequential Granted XID : {102377, S} , {104576, S} . The selected victim is XID : 104576. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:89) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Util.java:256) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:424) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:82) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(EmbedPreparedStatement.java:151) at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(EmbedPreparedStatement20.java:82) at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(EmbedPreparedStatement30.java:63) at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.<init>(EmbedPreparedStatement40.java:40) at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Driver40.java:107) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(EmbedConnection.java:1615) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(EmbedConnection.java:1443) at com.idelji.operations.iasset.plugin.imon.dao.ProcessDAO.insertData(ProcessDAO.java:86) at com.idelji.operations.iasset.plugin.imon.comm.CPUProcessDataStoreWriter.onData(CPUProcessDataStoreWriter.java:56) at com.idelji.operations.iasset.plugin.imon.handler.CPUProcessDataHandler.gotData(CPUProcessDataHandler.java:205) at com.idelji.operations.ibase.core.comm.Connector.makeRequest(Connector.java:83) at com.idelji.operations.ibase.core.comm.EntityConnector.makeRequest(EntityConnector.java:36) at com.idelji.operations.iasset.plugin.imon.comm.CPUProcessCommunicator.run(CPUProcessCommunicator.java:98) ***********************************************************************************************************************************************************
          Hide
          Saurabh Singhi added a comment -

          Hi Rick,
          The "something" that went wrong was the same container related exception that this thread is all about. And the new experiment that I'm soon going to try out would be just as you suggested.
          Thanks.
          Saurabh

          Show
          Saurabh Singhi added a comment - Hi Rick, The "something" that went wrong was the same container related exception that this thread is all about. And the new experiment that I'm soon going to try out would be just as you suggested. Thanks. Saurabh
          Hide
          Rick Hillegas added a comment -

          Hi Saurabh,

          Just want to make sure we're not talking past one another. The experiment (using a new, 10.8.2.3 database) would be to LOCK the table before compressing the table with SYSCS_COMPRESS_TABLE. On 2012-05-12 you said that you tried that experiment and something went wrong. It would be good to understand what went wrong.

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Saurabh, Just want to make sure we're not talking past one another. The experiment (using a new, 10.8.2.3 database) would be to LOCK the table before compressing the table with SYSCS_COMPRESS_TABLE. On 2012-05-12 you said that you tried that experiment and something went wrong. It would be good to understand what went wrong. Thanks, -Rick
          Hide
          Saurabh Singhi added a comment -

          Hi Rick,
          I tried locking the corrupted table but it did not yield any different results. Should I include a LOCK statement with the tables created and populated with the new 10.8 trunk and inform you of the results?

          Show
          Saurabh Singhi added a comment - Hi Rick, I tried locking the corrupted table but it did not yield any different results. Should I include a LOCK statement with the tables created and populated with the new 10.8 trunk and inform you of the results?
          Hide
          Rick Hillegas added a comment -

          Thanks, Saurabh. Can you describe what went wrong when you locked the table before compressing it? Thanks.

          Show
          Rick Hillegas added a comment - Thanks, Saurabh. Can you describe what went wrong when you locked the table before compressing it? Thanks.
          Hide
          Saurabh Singhi added a comment -

          Yes Rick, I have tried acquiring an exclusive lock before issuing the SYSCS_COMPRESS_TABLE, but that did not quite work either.
          Thanks,
          Saurabh

          Show
          Saurabh Singhi added a comment - Yes Rick, I have tried acquiring an exclusive lock before issuing the SYSCS_COMPRESS_TABLE, but that did not quite work either. Thanks, Saurabh
          Hide
          Rick Hillegas added a comment -

          Thanks, Saurabh. I see that DERBY-4275 is fixed in the trunk and in the 10.8 branch. DERBY-5351 was closed as a duplicate of DERBY-4275. However, it looks like DERBY-3683 and DERBY-5358 are still open.

          Have you tried issuing a LOCK TABLE command to seize exclusive access to the table before you issue SYSCS_COMPRESS_TABLE? This will throttle your concurrency every time you compress the table, but that is better than corrupting the table.

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Thanks, Saurabh. I see that DERBY-4275 is fixed in the trunk and in the 10.8 branch. DERBY-5351 was closed as a duplicate of DERBY-4275 . However, it looks like DERBY-3683 and DERBY-5358 are still open. Have you tried issuing a LOCK TABLE command to seize exclusive access to the table before you issue SYSCS_COMPRESS_TABLE? This will throttle your concurrency every time you compress the table, but that is better than corrupting the table. Thanks, -Rick
          Hide
          Saurabh Singhi added a comment -

          Rick,
          I was using SYSCS_COMPRESS_TABLE originally, but that lead to: DERBY-4275,DERBY-5351, DERBY-3683 & DERBY-5358, so I had to switch to the INPLACE version.
          Thanks,
          Saurabh

          Show
          Saurabh Singhi added a comment - Rick, I was using SYSCS_COMPRESS_TABLE originally, but that lead to: DERBY-4275 , DERBY-5351 , DERBY-3683 & DERBY-5358 , so I had to switch to the INPLACE version. Thanks, Saurabh
          Hide
          Rick Hillegas added a comment -

          Hi Saurabh,

          Have you tried running SYSCS_COMPRESS_TABLE rather than SYSCS_INPLACE_COMPRESS_TABLE? On 2011-05-18, Varma reported that SYSCS_COMPRESS_TABLE improved the situation and may have fixed the corrupted table.

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Saurabh, Have you tried running SYSCS_COMPRESS_TABLE rather than SYSCS_INPLACE_COMPRESS_TABLE? On 2011-05-18, Varma reported that SYSCS_COMPRESS_TABLE improved the situation and may have fixed the corrupted table. Thanks, -Rick
          Hide
          Saurabh Singhi added a comment - - edited

          Hi Rick,
          I created and ran an ij script against my database, but that does not reproduce the problem.
          I'm not really sure what else to do here.
          Thanks,
          Saurabh.

          Show
          Saurabh Singhi added a comment - - edited Hi Rick, I created and ran an ij script against my database, but that does not reproduce the problem. I'm not really sure what else to do here. Thanks, Saurabh.
          Hide
          Saurabh Singhi added a comment -

          Hi Rick,

          Actually, I did not know about an ij script until you just mentioned. I'll create one and upload here. Also, I'll try to isolate the code that is making the calls and possibly upload that too here along with my db.

          One problem in doing so is that the application is doing delete, compress and insert operations simultaneously and it might be hard to separate the code, but I'll see if I can do that so that the bug can be reproduced at your end.

          Also, I did try calling the INPLACE command with all combinations and it did not have any effect.

          Thanks.
          Saurabh

          Show
          Saurabh Singhi added a comment - Hi Rick, Actually, I did not know about an ij script until you just mentioned. I'll create one and upload here. Also, I'll try to isolate the code that is making the calls and possibly upload that too here along with my db. One problem in doing so is that the application is doing delete, compress and insert operations simultaneously and it might be hard to separate the code, but I'll see if I can do that so that the bug can be reproduced at your end. Also, I did try calling the INPLACE command with all combinations and it did not have any effect. Thanks. Saurabh
          Hide
          Rick Hillegas added a comment -

          Hi Saurabh,

          By scripting, I mean: can you attach a program or ij script which causes this corruption? That will allow us to reproduce the problem in the laboratory.

          Have you considered setting the other trailing arguments of the compression procedure, like so:

          CALL SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE('APP', 'TABLENAME', 1, 1, 1)

          This will cause the compression to take longer because Derby will do a more thorough job of garbage-collecting the empty space. That may take Derby down code paths which work around this bug. It would be interesting to know whether the bug occurs when you specify this extra garbage collection. To run this experiment you will again need to start with a new, uncorrupted database created by 10.8.2.3.

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Saurabh, By scripting, I mean: can you attach a program or ij script which causes this corruption? That will allow us to reproduce the problem in the laboratory. Have you considered setting the other trailing arguments of the compression procedure, like so: CALL SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE('APP', 'TABLENAME', 1, 1, 1) This will cause the compression to take longer because Derby will do a more thorough job of garbage-collecting the empty space. That may take Derby down code paths which work around this bug. It would be interesting to know whether the bug occurs when you specify this extra garbage collection. To run this experiment you will again need to start with a new, uncorrupted database created by 10.8.2.3. Thanks, -Rick
          Hide
          Saurabh Singhi added a comment -

          Hi Rick,

          I still cannot get past the problem.

          I'm using the command : CALL SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE('APP', 'TABLENAME', 0, 0, 1) .

          I don't quite understand what do you mean by "scripting" it? Please elaborate.

          Thanks.
          -Saurabh

          Show
          Saurabh Singhi added a comment - Hi Rick, I still cannot get past the problem. I'm using the command : CALL SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE('APP', 'TABLENAME', 0, 0, 1) . I don't quite understand what do you mean by "scripting" it? Please elaborate. Thanks. -Saurabh
          Hide
          Rick Hillegas added a comment -

          Hi Saurabh,

          Are you able to script the problem you are seeing? I am unable to reproduce it. Both of the repros attached to this issue now run correctly for me, even when I boost the number of rows by a factor of 10.

          What data compression command are you using? The repros use this command:

          CALL SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE('APP', 'TABLENAME', 0, 0, 1)

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Saurabh, Are you able to script the problem you are seeing? I am unable to reproduce it. Both of the repros attached to this issue now run correctly for me, even when I boost the number of rows by a factor of 10. What data compression command are you using? The repros use this command: CALL SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE('APP', 'TABLENAME', 0, 0, 1) Thanks, -Rick
          Hide
          Saurabh Singhi added a comment -

          Hi Rick,
          I'm running 10.8.2.3 (rev. 1338623). The tables were created using this revision and the problem still persists.
          Thanks.
          Best,
          Saurabh

          Show
          Saurabh Singhi added a comment - Hi Rick, I'm running 10.8.2.3 (rev. 1338623). The tables were created using this revision and the problem still persists. Thanks. Best, Saurabh
          Hide
          Rick Hillegas added a comment -

          Hi Saurabh,

          Derby5234.java is just a test case. It is not checked into the Derby codeline anywhere. You have the patched version of 10.8.2.3 provided that the revision stamp on derby.jar is bigger than revision 1337258. What do you see when you run sysinfo on your 10.8.2.3 installation:

          java -jar lib/derbyrun.jar sysinfo

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Saurabh, Derby5234.java is just a test case. It is not checked into the Derby codeline anywhere. You have the patched version of 10.8.2.3 provided that the revision stamp on derby.jar is bigger than revision 1337258. What do you see when you run sysinfo on your 10.8.2.3 installation: java -jar lib/derbyrun.jar sysinfo Thanks, -Rick
          Hide
          Saurabh Singhi added a comment -

          Hi Brett,
          Actually, I do have the "Limit rows to 100" option disabled, so the DB is returning me all the rows.

          Show
          Saurabh Singhi added a comment - Hi Brett, Actually, I do have the "Limit rows to 100" option disabled, so the DB is returning me all the rows.
          Hide
          Saurabh Singhi added a comment - - edited

          Hi Rick,

          Actually, I ran my application with version 10.8.2.3 (before your changes to Derby5234.java), the tables did get created with 10.8.2.3 but the problem reoccurred.

          Do you think running it with the latest changes(post Derby5234.java revision), would make any difference?

          Thanks.
          Best,
          Saurabh

          Show
          Saurabh Singhi added a comment - - edited Hi Rick, Actually, I ran my application with version 10.8.2.3 (before your changes to Derby5234.java), the tables did get created with 10.8.2.3 but the problem reoccurred. Do you think running it with the latest changes(post Derby5234.java revision), would make any difference? Thanks. Best, Saurabh
          Hide
          Rick Hillegas added a comment -

          Hi Saurabh,

          The successful completion of the SELECT COUNT is a good sign. At this point, I would try to dump the old, corrupted database and load it into a new 10.8.2.3 database.

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Saurabh, The successful completion of the SELECT COUNT is a good sign. At this point, I would try to dump the old, corrupted database and load it into a new 10.8.2.3 database. Thanks, -Rick
          Hide
          Brett Bergquist added a comment -

          Hi Saurabh,,

          Could it be the you have the SquirrelSQL client option turned off that retrieves only a limited set of rows (by default 100) and are returning many rows which is then causing the client to crash with the OutOfMemory and not the Derby engine crashing?

          Just a thought as I know that I have caused SquirrelSQL to do this when I have turned this option off and retrieved 10's of thousands of rows. The tool tries to create a Swing table to display the rows and this causes the OutOfMemory condition in the tool, not the database engine.

          Brett

          Show
          Brett Bergquist added a comment - Hi Saurabh,, Could it be the you have the SquirrelSQL client option turned off that retrieves only a limited set of rows (by default 100) and are returning many rows which is then causing the client to crash with the OutOfMemory and not the Derby engine crashing? Just a thought as I know that I have caused SquirrelSQL to do this when I have turned this option off and retrieved 10's of thousands of rows. The tool tries to create a Swing table to display the rows and this causes the OutOfMemory condition in the tool, not the database engine. Brett
          Hide
          Saurabh Singhi added a comment -

          Hi Rick,
          The count query gives me 626677 rows.

          Show
          Saurabh Singhi added a comment - Hi Rick, The count query gives me 626677 rows.
          Hide
          Rick Hillegas added a comment - - edited

          Hi Saurabh,

          Hard to say where the OOM is coming from. It may be that the Squirrel Client runs out of memory while trying to prepare a big window for display. What happens if you SELECT COUNT from the corrupted table? The VTI attached to DERBY-4962 will not try to materialize all of the results at once. You may be able to use it to stream the corrupted table into a new, empty database.

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - - edited Hi Saurabh, Hard to say where the OOM is coming from. It may be that the Squirrel Client runs out of memory while trying to prepare a big window for display. What happens if you SELECT COUNT from the corrupted table? The VTI attached to DERBY-4962 will not try to materialize all of the results at once. You may be able to use it to stream the corrupted table into a new, empty database. Thanks, -Rick
          Hide
          Saurabh Singhi added a comment -

          Hi Rick,

          When I try to do a Select * on the table (via Squirrel Client) that I suspect is corrupted, it gives me a "java.lang.OutOfMemoryError: Java heap space" due to the fact that "The query results have exceeded the maximum amount of memory allocated to this application.". The allocated memory seems to be 247 MB.
          Could that be a cause for the fix not working?

          Thanks,
          Saurabh

          Show
          Saurabh Singhi added a comment - Hi Rick, When I try to do a Select * on the table (via Squirrel Client) that I suspect is corrupted, it gives me a "java.lang.OutOfMemoryError: Java heap space" due to the fact that "The query results have exceeded the maximum amount of memory allocated to this application.". The allocated memory seems to be 247 MB. Could that be a cause for the fix not working? Thanks, Saurabh
          Hide
          Rick Hillegas added a comment -

          Hi Saurabh,

          There is no additional fix. You just need a build from the 10.8 branch after revision 1337258.

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Saurabh, There is no additional fix. You just need a build from the 10.8 branch after revision 1337258. Thanks, -Rick
          Hide
          Saurabh Singhi added a comment -

          Hi Rick,
          Should I get the latest branch again? Would it have this fix?

          Show
          Saurabh Singhi added a comment - Hi Rick, Should I get the latest branch again? Would it have this fix?
          Hide
          Rick Hillegas added a comment -

          Hi Saurabh,

          Can you SELECT all of the rows in the corrupted table? I find that the table can still be read after the corruption when I run my toy test case Derby5234.

          If you can SELECT all of the rows from the corrupted table, then I suggest the following course of action:

          A) Using 10.8.2.3, create a new database.

          B) Copy all of the data from the corrupted database into the new database. You can use the dump and bulk load procedures to do this. Alternatively, you can use the schema-importing VTI attached to DERBY-4962. The latter approach may be faster.

          C) Delete the old, corrupted database or rename it to make it clear that it should not be used in production anymore.

          D) Rename the new database to be the old name of the corrupted database.

          E) Continue running your application with 10.8.2.3 and the new (now renamed) database.

          Hope this helps,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Saurabh, Can you SELECT all of the rows in the corrupted table? I find that the table can still be read after the corruption when I run my toy test case Derby5234. If you can SELECT all of the rows from the corrupted table, then I suggest the following course of action: A) Using 10.8.2.3, create a new database. B) Copy all of the data from the corrupted database into the new database. You can use the dump and bulk load procedures to do this. Alternatively, you can use the schema-importing VTI attached to DERBY-4962 . The latter approach may be faster. C) Delete the old, corrupted database or rename it to make it clear that it should not be used in production anymore. D) Rename the new database to be the old name of the corrupted database. E) Continue running your application with 10.8.2.3 and the new (now renamed) database. Hope this helps, -Rick
          Hide
          Rick Hillegas added a comment -

          Attaching a revised version of Derby5234.java. This version successfully selects all of the rows from the corrupted table. After the corruption occurs, the repro now does the following:

          1) Brings down the Derby engine.

          2) Gets a new connection to the database.

          3) Reads all of the rows in the table.

          Show
          Rick Hillegas added a comment - Attaching a revised version of Derby5234.java. This version successfully selects all of the rows from the corrupted table. After the corruption occurs, the repro now does the following: 1) Brings down the Derby engine. 2) Gets a new connection to the database. 3) Reads all of the rows in the table.
          Hide
          Saurabh Singhi added a comment - - edited

          Hi Rick,

          1) It is an existing/corrupted data table that I ran the fix against.
          It seems like after a certain limit (~1.3 gb), the database starts acting fuzzy and gets corrupted. Our (multi-threaded) application continually inserts data into the tables and periodically (once a day) deletes rows from the previous day and after the delete command, I am using the compress call to squeeze the db back into shape. Originally, I used the 'SYSCS_UTIL.SYSCS_COMPRESS_TABLE' call, but that lead to: DERBY-4275,DERBY-5351, DERBY-3683 & DERBY-5358.
          Following instructions, I used the 'INPLACE' call and was able to get rid of those 2 problems but now I'm stuck with this one.

          2) Before I try to insert new data, I'm deleting (and compressing) the old (corrupted) data and if the corrupted data is being deleted, then why do I still get the error?

          3) This only started happening after I included the COMPRESS_TABLE calls. Prior to that, even though it didn't release the deleted rows' space back to the OS, but it did not give errors.

          Thanks.
          Best,
          Saurabh

          Show
          Saurabh Singhi added a comment - - edited Hi Rick, 1) It is an existing/corrupted data table that I ran the fix against. It seems like after a certain limit (~1.3 gb), the database starts acting fuzzy and gets corrupted. Our (multi-threaded) application continually inserts data into the tables and periodically (once a day) deletes rows from the previous day and after the delete command, I am using the compress call to squeeze the db back into shape. Originally, I used the 'SYSCS_UTIL.SYSCS_COMPRESS_TABLE' call, but that lead to: DERBY-4275 , DERBY-5351 , DERBY-3683 & DERBY-5358 . Following instructions, I used the 'INPLACE' call and was able to get rid of those 2 problems but now I'm stuck with this one. 2) Before I try to insert new data, I'm deleting (and compressing) the old (corrupted) data and if the corrupted data is being deleted, then why do I still get the error? 3) This only started happening after I included the COMPRESS_TABLE calls. Prior to that, even though it didn't release the deleted rows' space back to the OS, but it did not give errors. Thanks. Best, Saurabh
          Hide
          Rick Hillegas added a comment -

          Hi Saurabh,

          I just want to confirm whether your latest bug report was against a table which had been newly created with 10.8.2.3. Note the following:

          1) I observe that I can still reboot a database which was already corrupted by the original form of this bug. And I can successfully delete all rows in the corrupted table and I can run inplace compress on the table.

          2) The patch does not fix an already corrupted table. It only prevents a previously uncorrupted table from being corrupted this way.

          So I just want to confirm: Did you newly recreate the table using 10.8.2.3? Or was the table already corrupted and you were hoping that the patch would fix the broken table?

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Saurabh, I just want to confirm whether your latest bug report was against a table which had been newly created with 10.8.2.3. Note the following: 1) I observe that I can still reboot a database which was already corrupted by the original form of this bug. And I can successfully delete all rows in the corrupted table and I can run inplace compress on the table. 2) The patch does not fix an already corrupted table. It only prevents a previously uncorrupted table from being corrupted this way. So I just want to confirm: Did you newly recreate the table using 10.8.2.3? Or was the table already corrupted and you were hoping that the patch would fix the broken table? Thanks, -Rick
          Hide
          Rick Hillegas added a comment -

          Reopening this issue. Although the repro passes, the bug is not fixed in production.

          Show
          Rick Hillegas added a comment - Reopening this issue. Although the repro passes, the bug is not fixed in production.
          Hide
          Saurabh Singhi added a comment -

          Hi Rick,

          I built version 10.8.2.3, but unfortunately, I'm still getting the same errors. Here's a dump of the same (note that the container exception occurs a little deeper down the stack trace) :-

          *****************************************************************************************************************************************************************************
          1 May 2012 11:29 DataDictionary.deleteHRData = deleteCmd = DELETE FROM APP.CPU_059222_HR WHERE SampleTimestamp <= ?
          11 May 2012 11:29 DataDictionary.deleteHRData = Delete executed with Timestamp = Thu May 10 11:29:09 PDT 2012
          11 May 2012 11:29 DataDictionary.reclaimDiskSpace = compressCmd:->call SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE('APP', 'CPU_059222_HR',0,0, 1)
          11 May 2012 11:29 DataDictionary.reclaimDiskSpace = compressSqlStmt executed for Table:->CPU_059222_HR
          11 May 2012 11:29 DataDictionary.deleteHRData = Hourly retention = 24
          11 May 2012 11:29 DataDictionary.deleteHRData = deleteCmd = DELETE FROM APP.PROCESS_059222_HR WHERE SampleTimestamp <= ?
          11 May 2012 11:29 DataDictionary.deleteHRData = deleteHRData: SQL error - Java exception: ': java.lang.NullPointerException'.
          11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'.
          java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142)
          at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634)
          at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73)
          at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51)
          at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70)
          at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248)
          at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59)
          at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464)
          at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54)
          Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71)
          ... 16 more
          Caused by: java.lang.NullPointerException
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589)
          at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316)
          at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910)
          at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401)
          at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422)
          ... 10 more
          11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'.
          java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142)
          at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634)
          at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73)
          at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51)
          at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70)
          at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248)
          at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59)
          at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464)
          at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54)
          Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71)
          ... 16 more
          Caused by: java.lang.NullPointerException
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589)
          at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316)
          at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910)
          at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401)
          at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422)
          ... 10 more
          11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'.
          java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142)
          at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634)
          at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73)
          at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51)
          at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70)
          at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248)
          at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59)
          at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464)
          at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54)
          Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71)
          ... 16 more
          Caused by: java.lang.NullPointerException
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589)
          at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316)
          at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910)
          at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401)
          at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422)
          ... 10 more
          11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'.
          java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142)
          at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634)
          at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73)
          at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51)
          at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70)
          at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248)
          at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59)
          at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464)
          at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54)
          Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71)
          ... 16 more
          Caused by: java.lang.NullPointerException
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589)
          at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316)
          at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910)
          at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401)
          at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422)
          ... 10 more
          11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'.
          java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142)
          at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634)
          at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73)
          at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51)
          at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70)
          at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248)
          at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59)
          at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464)
          at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54)
          Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71)
          ... 16 more
          Caused by: java.lang.NullPointerException
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589)
          at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316)
          at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910)
          at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401)
          at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422)
          ... 10 more
          11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'.
          java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142)
          at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634)
          at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73)
          at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51)
          at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70)
          at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248)
          at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59)
          at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464)
          at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54)
          Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71)
          ... 16 more
          Caused by: java.lang.NullPointerException
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589)
          at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316)
          at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910)
          at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401)
          at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422)
          ... 10 more
          11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'.
          java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142)
          at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634)
          at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73)
          at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51)
          at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70)
          at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248)
          at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59)
          at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464)
          at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54)
          Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71)
          ... 16 more
          Caused by: java.lang.NullPointerException
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589)
          at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316)
          at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910)
          at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476)
          at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401)
          at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215)
          at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422)
          ... 10 more
          11 May 2012 11:29 ProcessDAO.insertData = Page Page(41494,Container(0, 2128)) could not be read from disk.
          java.sql.BatchUpdateException: Page Page(41494,Container(0, 2128)) could not be read from disk.
          at org.apache.derby.impl.jdbc.EmbedStatement.executeBatch(EmbedStatement.java:1008)
          at com.idelji.operations.iasset.plugin.imon.dao.ProcessDAO.insertData(ProcessDAO.java:142)
          at com.idelji.operations.iasset.plugin.imon.comm.CPUProcessDataStoreWriter.onData(CPUProcessDataStoreWriter.java:56)
          at com.idelji.operations.iasset.plugin.imon.handler.CPUProcessDataHandler.gotData(CPUProcessDataHandler.java:205)
          at com.idelji.operations.ibase.core.comm.Connector.makeRequest(Connector.java:83)
          at com.idelji.operations.ibase.core.comm.EntityConnector.makeRequest(EntityConnector.java:36)
          at com.idelji.operations.iasset.plugin.imon.comm.CPUProcessCommunicator.run(CPUProcessCommunicator.java:98)
          Caused by: java.sql.SQLException: Page Page(41494,Container(0, 2128)) could not be read from disk.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142)
          at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:278)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:431)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290)
          at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:82)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1334)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeBatchElement(EmbedPreparedStatement.java:1040)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeBatch(EmbedStatement.java:984)
          ... 6 more
          Caused by: java.sql.SQLException: Page Page(41494,Container(0, 2128)) could not be read from disk.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71)
          ... 15 more
          Caused by: java.sql.SQLException: Java exception: 'Reached end of file while attempting to read a whole page.: java.io.EOFException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142)
          at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436)
          ... 13 more
          Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:1165)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:429)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:369)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:263)
          at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:673)
          at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:193)
          at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295)
          at org.apache.derby.impl.store.raw.data.FileContainer.initPage(FileContainer.java:2431)
          at org.apache.derby.impl.store.raw.data.FileContainer.newPage(FileContainer.java:1879)
          at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(BaseContainer.java:314)
          at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(BaseContainerHandle.java:183)
          at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:302)
          at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599)
          at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452)
          at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1028)
          at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:505)
          at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:438)
          at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:319)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1242)
          ... 8 more
          *****************************************************************************************************************************************************************************

          I could give you my database, if you need to run your own tests against it.
          Best,
          Saurabh

          Show
          Saurabh Singhi added a comment - Hi Rick, I built version 10.8.2.3, but unfortunately, I'm still getting the same errors. Here's a dump of the same (note that the container exception occurs a little deeper down the stack trace) :- ***************************************************************************************************************************************************************************** 1 May 2012 11:29 DataDictionary.deleteHRData = deleteCmd = DELETE FROM APP.CPU_059222_HR WHERE SampleTimestamp <= ? 11 May 2012 11:29 DataDictionary.deleteHRData = Delete executed with Timestamp = Thu May 10 11:29:09 PDT 2012 11 May 2012 11:29 DataDictionary.reclaimDiskSpace = compressCmd:->call SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE('APP', 'CPU_059222_HR',0,0, 1) 11 May 2012 11:29 DataDictionary.reclaimDiskSpace = compressSqlStmt executed for Table:->CPU_059222_HR 11 May 2012 11:29 DataDictionary.deleteHRData = Hourly retention = 24 11 May 2012 11:29 DataDictionary.deleteHRData = deleteCmd = DELETE FROM APP.PROCESS_059222_HR WHERE SampleTimestamp <= ? 11 May 2012 11:29 DataDictionary.deleteHRData = deleteHRData: SQL error - Java exception: ': java.lang.NullPointerException'. 11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'. java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142) at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634) at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73) at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51) at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70) at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248) at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144) at java.sql.DriverManager.getConnection(Unknown Source) at java.sql.DriverManager.getConnection(Unknown Source) at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59) at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464) at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54) Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71) ... 16 more Caused by: java.lang.NullPointerException at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589) at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316) at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910) at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401) at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295) at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422) ... 10 more 11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'. java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142) at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634) at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73) at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51) at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70) at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248) at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144) at java.sql.DriverManager.getConnection(Unknown Source) at java.sql.DriverManager.getConnection(Unknown Source) at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59) at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464) at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54) Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71) ... 16 more Caused by: java.lang.NullPointerException at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589) at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316) at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910) at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401) at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295) at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422) ... 10 more 11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'. java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142) at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634) at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73) at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51) at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70) at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248) at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144) at java.sql.DriverManager.getConnection(Unknown Source) at java.sql.DriverManager.getConnection(Unknown Source) at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59) at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464) at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54) Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71) ... 16 more Caused by: java.lang.NullPointerException at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589) at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316) at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910) at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401) at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295) at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422) ... 10 more 11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'. java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142) at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634) at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73) at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51) at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70) at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248) at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144) at java.sql.DriverManager.getConnection(Unknown Source) at java.sql.DriverManager.getConnection(Unknown Source) at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59) at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464) at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54) Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71) ... 16 more Caused by: java.lang.NullPointerException at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589) at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316) at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910) at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401) at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295) at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422) ... 10 more 11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'. java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142) at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634) at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73) at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51) at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70) at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248) at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144) at java.sql.DriverManager.getConnection(Unknown Source) at java.sql.DriverManager.getConnection(Unknown Source) at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59) at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464) at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54) Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71) ... 16 more Caused by: java.lang.NullPointerException at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589) at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316) at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910) at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401) at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295) at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422) ... 10 more 11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'. java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142) at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634) at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73) at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51) at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70) at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248) at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144) at java.sql.DriverManager.getConnection(Unknown Source) at java.sql.DriverManager.getConnection(Unknown Source) at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59) at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464) at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54) Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71) ... 16 more Caused by: java.lang.NullPointerException at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589) at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316) at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910) at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401) at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295) at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422) ... 10 more 11 May 2012 11:29 SchemaDAO.tableExists = Java exception: ': java.lang.NullPointerException'. java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142) at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:634) at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73) at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51) at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70) at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248) at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144) at java.sql.DriverManager.getConnection(Unknown Source) at java.sql.DriverManager.getConnection(Unknown Source) at com.idelji.operations.iasset.plugin.imon.dao.SchemaDAO.tableExists(SchemaDAO.java:59) at com.idelji.operations.iasset.plugin.imon.dao.DataDictionary.deleteHRData(DataDictionary.java:464) at com.idelji.operations.iasset.plugin.imon.service.CleanUpTask.run(CleanUpTask.java:54) Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71) ... 16 more Caused by: java.lang.NullPointerException at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:659) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:589) at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1316) at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:910) at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:689) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:476) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1308) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(DataDictionaryImpl.java:9271) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:9229) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1722) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1589) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:425) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:401) at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:295) at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:189) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(EmbedConnection.java:1251) at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(EmbedConnection.java:1215) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:422) ... 10 more 11 May 2012 11:29 ProcessDAO.insertData = Page Page(41494,Container(0, 2128)) could not be read from disk. java.sql.BatchUpdateException: Page Page(41494,Container(0, 2128)) could not be read from disk. at org.apache.derby.impl.jdbc.EmbedStatement.executeBatch(EmbedStatement.java:1008) at com.idelji.operations.iasset.plugin.imon.dao.ProcessDAO.insertData(ProcessDAO.java:142) at com.idelji.operations.iasset.plugin.imon.comm.CPUProcessDataStoreWriter.onData(CPUProcessDataStoreWriter.java:56) at com.idelji.operations.iasset.plugin.imon.handler.CPUProcessDataHandler.gotData(CPUProcessDataHandler.java:205) at com.idelji.operations.ibase.core.comm.Connector.makeRequest(Connector.java:83) at com.idelji.operations.ibase.core.comm.EntityConnector.makeRequest(EntityConnector.java:36) at com.idelji.operations.iasset.plugin.imon.comm.CPUProcessCommunicator.run(CPUProcessCommunicator.java:98) Caused by: java.sql.SQLException: Page Page(41494,Container(0, 2128)) could not be read from disk. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142) at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:278) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:431) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2290) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:82) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1334) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeBatchElement(EmbedPreparedStatement.java:1040) at org.apache.derby.impl.jdbc.EmbedStatement.executeBatch(EmbedStatement.java:984) ... 6 more Caused by: java.sql.SQLException: Page Page(41494,Container(0, 2128)) could not be read from disk. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71) ... 15 more Caused by: java.sql.SQLException: Java exception: 'Reached end of file while attempting to read a whole page.: java.io.EOFException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142) at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436) ... 13 more Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page. at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:1165) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:429) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:369) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:263) at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:673) at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:193) at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295) at org.apache.derby.impl.store.raw.data.FileContainer.initPage(FileContainer.java:2431) at org.apache.derby.impl.store.raw.data.FileContainer.newPage(FileContainer.java:1879) at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(BaseContainer.java:314) at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(BaseContainerHandle.java:183) at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:302) at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599) at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452) at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1028) at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:505) at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:438) at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:319) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1242) ... 8 more ***************************************************************************************************************************************************************************** I could give you my database, if you need to run your own tests against it. Best, Saurabh
          Hide
          Rick Hillegas added a comment -

          Resolving this issue now that the fix has been backported to 10.8. Follow-on work was discussed on 2012-05-04. That work can continue under DERBY-5761.

          Show
          Rick Hillegas added a comment - Resolving this issue now that the fix has been backported to 10.8. Follow-on work was discussed on 2012-05-04. That work can continue under DERBY-5761 .
          Hide
          Rick Hillegas added a comment -

          Hi Saurabh,

          The 10.9 development trunk contains what we think is a fix for this bug. I have just ported that fix to the 10.8 branch.

          The fix should appear in the 10.9 feature release which we will evaluate later this month: http://wiki.apache.org/db-derby/DerbyTenNineOneRelease

          There is also talk about producing a 10.8.3 maintenance release after the community publishes a 10.9 feature release. That maintenance release will also contain the fix.

          Hope this helps,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Saurabh, The 10.9 development trunk contains what we think is a fix for this bug. I have just ported that fix to the 10.8 branch. The fix should appear in the 10.9 feature release which we will evaluate later this month: http://wiki.apache.org/db-derby/DerbyTenNineOneRelease There is also talk about producing a 10.8.3 maintenance release after the community publishes a 10.9 feature release. That maintenance release will also contain the fix. Hope this helps, -Rick
          Hide
          Rick Hillegas added a comment -

          Ported 1335570 and 1335677 from trunk to 10.8 branch at subversion revision 1337258.

          Show
          Rick Hillegas added a comment - Ported 1335570 and 1335677 from trunk to 10.8 branch at subversion revision 1337258.
          Hide
          Saurabh Singhi added a comment -

          I am getting the same error, even after calling SYSCS_UTIL.SYSCS_COMPRESS_TABLE (or SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE ) programatically, before and/or after the DELETE statement. The DB size is 1.3 gb.
          Any updates on when a quick-fix or workaround or the actual fix would be available? This is an urgent requirement. Thanks.

          Show
          Saurabh Singhi added a comment - I am getting the same error, even after calling SYSCS_UTIL.SYSCS_COMPRESS_TABLE (or SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE ) programatically, before and/or after the DELETE statement. The DB size is 1.3 gb. Any updates on when a quick-fix or workaround or the actual fix would be available? This is an urgent requirement. Thanks.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-5234-02-aa-edgeCaseTests.diff. This adds a couple more edge cases to the test. Committed at subversion revision 1335677.

          Show
          Rick Hillegas added a comment - Attaching derby-5234-02-aa-edgeCaseTests.diff. This adds a couple more edge cases to the test. Committed at subversion revision 1335677.
          Hide
          Rick Hillegas added a comment -

          Tests passed cleanly for me. Committed derby-5234-01-ab-emptyAllocPage.diff at subversion revision 1335570.

          Show
          Rick Hillegas added a comment - Tests passed cleanly for me. Committed derby-5234-01-ab-emptyAllocPage.diff at subversion revision 1335570.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-5234-01-ab-emptyAllocPage.diff. This adds a regression test case to the previous rev of the patch. The test runs correctly with the patch and fails with an EOFException without the patch. I am running full regression tests again.

          Touches the following additional files:

          M java/testing/org/apache/derbyTesting/functionTests/tests/store/_Suite.java
          A java/testing/org/apache/derbyTesting/functionTests/tests/store/Derby5234Test.java

          Show
          Rick Hillegas added a comment - Attaching derby-5234-01-ab-emptyAllocPage.diff. This adds a regression test case to the previous rev of the patch. The test runs correctly with the patch and fails with an EOFException without the patch. I am running full regression tests again. Touches the following additional files: M java/testing/org/apache/derbyTesting/functionTests/tests/store/_Suite.java A java/testing/org/apache/derbyTesting/functionTests/tests/store/Derby5234Test.java
          Hide
          Rick Hillegas added a comment -

          Attaching Derby5234.java, a simpler, faster version of the repro. On my machine, this repro trips the bug in about 12 seconds. I will probably use this simpler code in the test case which will accompany the first patch.

          Show
          Rick Hillegas added a comment - Attaching Derby5234.java, a simpler, faster version of the repro. On my machine, this repro trips the bug in about 12 seconds. I will probably use this simpler code in the test case which will accompany the first patch.
          Hide
          Rick Hillegas added a comment -

          Thanks, Mike. I will add a test case to the patch. Please add comments on top of the patch once I've checked it in--I think that your comments will be more useful than mine. I will also put some thought into writing a follow-on patch containing tests for edge cases. Thanks.

          Show
          Rick Hillegas added a comment - Thanks, Mike. I will add a test case to the patch. Please add comments on top of the patch once I've checked it in--I think that your comments will be more useful than mine. I will also put some thought into writing a follow-on patch containing tests for edge cases. Thanks.
          Hide
          Mike Matrigali added a comment -

          the patch changes look right to me. the existing code (not just your changes) could use more comments, which probably led to the bugs in the first place. I like the changes to
          var names to make it clear they are bit positions of the pages not page numbers. I would be happy to add comments either after you checkin or produce a new patch based on your work, let me know how you want to procede.

          Definitely would be good to get a test case checked in with the patch.

          longer term - does not have to be part of this fix, trying to think of some complete edge case test, as at least fix for 1 is an off by one error. probably not for running every night. Something like
          2k page heap,
          for (i up to some number bigger than a few extents)
          drop/create table each time
          add rows such that it fills i pages
          delete all rows
          inplace compress
          some set of tests, at least insert i + n rows

          Show
          Mike Matrigali added a comment - the patch changes look right to me. the existing code (not just your changes) could use more comments, which probably led to the bugs in the first place. I like the changes to var names to make it clear they are bit positions of the pages not page numbers. I would be happy to add comments either after you checkin or produce a new patch based on your work, let me know how you want to procede. Definitely would be good to get a test case checked in with the patch. longer term - does not have to be part of this fix, trying to think of some complete edge case test, as at least fix for 1 is an off by one error. probably not for running every night. Something like 2k page heap, for (i up to some number bigger than a few extents) drop/create table each time add rows such that it fills i pages delete all rows inplace compress some set of tests, at least insert i + n rows
          Hide
          Mike Matrigali added a comment -

          Concerning the 3rd point above I have logged DERBY-5738. looks like intent was to handle multiple extents but at least in 10.1 and later the code does not handle shrinking past an extent.

          Show
          Mike Matrigali added a comment - Concerning the 3rd point above I have logged DERBY-5738 . looks like intent was to handle multiple extents but at least in 10.1 and later the code does not handle shrinking past an extent.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-5234-01-aa-emptyAllocPage.diff. These small changes make the repro run correctly. Regression tests pass cleanly on this patch.

          I have stumbled across at least 3 separate problems in the compression code. However, that may simply mean that I don't understand the code. The 3 problems are:

          1) A boundary checking error which causes an allocation extent to think that it still has pages, even though those pages have been released to the operating system. This is what causes the repro to fail.

          2) A confusion about whether a variable represents a bit position or a page number. This causes the code to not understand that all of the pages in an extent have been released. Fixing this check does not change any user-visible behavior, but I think that fixing the check is a step in the right direction.

          3) The inability of the compression code to release pages held by the first allocation page. I don't understand this problem yet. Before looking into this one, I need advice about whether I am heading in the right direction.

          More information about these 3 problems follows:

          -------------------

          Concerning (1), the boundary check which causes the repro to fail:

          In AllocExtent.compressPages(), the new_highest_page argument can be -1. This happens if all of the pages in the extent turn out to be empty. However, if new_highest_page is -1, then the code does not fall into the block at line 577; that's the code which actually marks the pages as released. The value of new_highest_page was calculated by AllocExtent.compress(). The variable name new_highest_page is a confusing name. This is a bit position and not a page number, and in the case when it is -1, it is a flag that all pages are empty. AllocExtent.compress() returns new_highest_page + 1, triggering its caller to fall into a block at line 1074 in AllocPage.compress(); that block releases pages to the operating system. That is how we end up in the situation that the pages are actually released but AllocExtent still thinks they are allocated. That, in turn, is what tricks a later INSERT into trying to write onto a non-existent page.

          The fix is to make the code fall into the block at 577 if new_highest_page is -1.

          -------------------

          Concerning (2), the confusion about whether AllocExtent.compress() returns a bit position or a page number:

          At line 1080 in AllocPage.compress(), the code compares a bit position to a page number. Bit positions are small integers, e.g., in the range 0-200. Page numbers are potentially larger integers in, say, the range 12000-12200. The weird comparison at line 1080 causes AllocPage.compress() to not recognize that all of the pages in the extent have been released.

          I have renamed last_valid_page to last_valid_page_bit to clarify that this is a bit position, not a page number. And I have changed the check at 1080 to compare the bit position to another bit position. This comparison deserves the attention of someone who knows this code better than I do. Is this the right comparison?

          In a follow-on cleanup issue, it might make sense to change variable names in the allocation code to clarify what is a bit number and what is a page number. This may disclose other questionable code in this area.

          -------------------

          Concerning (3), the inability of the compression code to release empty pages managed by the first allocation page:

          I had hoped that the change for (2) would cause the compress to release more space. But it didn't. The compress only releases the pages managed by the second (last) allocation page. All of the pages managed by the first allocation page are also empty, but they are not released. This seems wrong to me. I would expect the file to shrink back to its initial size.

          Before pursuing this follow-on issue, I would like advice about whether I am headed in the right direction. Should the compress shrink the page back to its initial size? Or should SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE('APP', 'OPERATIONS', 0, 0, 1) just release empty pages managed by the second and higher allocation pages?

          -------------------

          Touches the following files:

          M java/engine/org/apache/derby/impl/store/raw/data/AllocExtent.java

          Fix for (1).

          ----------

          M java/engine/org/apache/derby/impl/store/raw/data/AllocPage.java

          Clarification for (2).

          Show
          Rick Hillegas added a comment - Attaching derby-5234-01-aa-emptyAllocPage.diff. These small changes make the repro run correctly. Regression tests pass cleanly on this patch. I have stumbled across at least 3 separate problems in the compression code. However, that may simply mean that I don't understand the code. The 3 problems are: 1) A boundary checking error which causes an allocation extent to think that it still has pages, even though those pages have been released to the operating system. This is what causes the repro to fail. 2) A confusion about whether a variable represents a bit position or a page number. This causes the code to not understand that all of the pages in an extent have been released. Fixing this check does not change any user-visible behavior, but I think that fixing the check is a step in the right direction. 3) The inability of the compression code to release pages held by the first allocation page. I don't understand this problem yet. Before looking into this one, I need advice about whether I am heading in the right direction. More information about these 3 problems follows: ------------------- Concerning (1), the boundary check which causes the repro to fail: In AllocExtent.compressPages(), the new_highest_page argument can be -1. This happens if all of the pages in the extent turn out to be empty. However, if new_highest_page is -1, then the code does not fall into the block at line 577; that's the code which actually marks the pages as released. The value of new_highest_page was calculated by AllocExtent.compress(). The variable name new_highest_page is a confusing name. This is a bit position and not a page number, and in the case when it is -1, it is a flag that all pages are empty. AllocExtent.compress() returns new_highest_page + 1, triggering its caller to fall into a block at line 1074 in AllocPage.compress(); that block releases pages to the operating system. That is how we end up in the situation that the pages are actually released but AllocExtent still thinks they are allocated. That, in turn, is what tricks a later INSERT into trying to write onto a non-existent page. The fix is to make the code fall into the block at 577 if new_highest_page is -1. ------------------- Concerning (2), the confusion about whether AllocExtent.compress() returns a bit position or a page number: At line 1080 in AllocPage.compress(), the code compares a bit position to a page number. Bit positions are small integers, e.g., in the range 0-200. Page numbers are potentially larger integers in, say, the range 12000-12200. The weird comparison at line 1080 causes AllocPage.compress() to not recognize that all of the pages in the extent have been released. I have renamed last_valid_page to last_valid_page_bit to clarify that this is a bit position, not a page number. And I have changed the check at 1080 to compare the bit position to another bit position. This comparison deserves the attention of someone who knows this code better than I do. Is this the right comparison? In a follow-on cleanup issue, it might make sense to change variable names in the allocation code to clarify what is a bit number and what is a page number. This may disclose other questionable code in this area. ------------------- Concerning (3), the inability of the compression code to release empty pages managed by the first allocation page: I had hoped that the change for (2) would cause the compress to release more space. But it didn't. The compress only releases the pages managed by the second (last) allocation page. All of the pages managed by the first allocation page are also empty, but they are not released. This seems wrong to me. I would expect the file to shrink back to its initial size. Before pursuing this follow-on issue, I would like advice about whether I am headed in the right direction. Should the compress shrink the page back to its initial size? Or should SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE('APP', 'OPERATIONS', 0, 0, 1) just release empty pages managed by the second and higher allocation pages? ------------------- Touches the following files: M java/engine/org/apache/derby/impl/store/raw/data/AllocExtent.java Fix for (1). ---------- M java/engine/org/apache/derby/impl/store/raw/data/AllocPage.java Clarification for (2).
          Hide
          Rick Hillegas added a comment - - edited

          Might be a red herring, but I see that there is only one ChainAllocPageOperation in the transaction logs. That occurs in log85.dat. It allocates page 10217 (2 pages before the bad page) and chains it to page 0. I would expect to see a second ChainAllocPageOperation after the compress. But I don't.

          Here is the sequence of operations from log85.dat:

          begin transaction
          insert into slot 31 of page 10216
          end transaction
          checksum
          begin transaction
          initialize page 10217
          chain alloc page 10217 to page 0

          Here is the corresponding sequence of operations from log191.dat

          begin transaction
          insert into slot 31 of page 10216
          end transaction
          checksum
          begin transaction
          initialize page 10218
          alloc page 10218 (linking it to 10217)

          Something asymmetric is happening around page 10217.

          Show
          Rick Hillegas added a comment - - edited Might be a red herring, but I see that there is only one ChainAllocPageOperation in the transaction logs. That occurs in log85.dat. It allocates page 10217 (2 pages before the bad page) and chains it to page 0. I would expect to see a second ChainAllocPageOperation after the compress. But I don't. Here is the sequence of operations from log85.dat: begin transaction insert into slot 31 of page 10216 end transaction checksum begin transaction initialize page 10217 chain alloc page 10217 to page 0 Here is the corresponding sequence of operations from log191.dat begin transaction insert into slot 31 of page 10216 end transaction checksum begin transaction initialize page 10218 alloc page 10218 (linking it to 10217) Something asymmetric is happening around page 10217.
          Hide
          Rick Hillegas added a comment -

          Attaching log85.dat and log191.dat.

          log85.dat is the log file which contains the first, successful allocation of the bad page and then inserts into it. log191 is the last file, where Derby crashed at the point where it should have been logging a checksum.

          I decoded these log files as follows, but the output is too big to be attached to this JIRA:

          java LogFileReader $logFile -v

          Here is the sequence of good operations from log85.dat:

          begin transaction
          insert into slot 31 of page 10218
          end transaction
          checksum
          begin transaction
          initialize page 10219

          Here is the concluding sequence of operations from log191.dat

          begin transaction
          insert into slot 31 of page 10218
          end transaction
          <end of log file, this is where Derby crashes>

          Show
          Rick Hillegas added a comment - Attaching log85.dat and log191.dat. log85.dat is the log file which contains the first, successful allocation of the bad page and then inserts into it. log191 is the last file, where Derby crashed at the point where it should have been logging a checksum. I decoded these log files as follows, but the output is too big to be attached to this JIRA: java LogFileReader $logFile -v Here is the sequence of good operations from log85.dat: begin transaction insert into slot 31 of page 10218 end transaction checksum begin transaction initialize page 10219 Here is the concluding sequence of operations from log191.dat begin transaction insert into slot 31 of page 10218 end transaction <end of log file, this is where Derby crashes>
          Hide
          Rick Hillegas added a comment -

          Attaching 5234_alloc.out, 5234_page_10219.out, and 5234_summary.out.

          I turned on log archiving by calling SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE_NOWAIT. This resulted in accumulating 191 transaction log files. Fortunately, this did not affect the bug. The bug continues to occur on exactly the same page at exactly the same point in the test.

          To examine the transaction logs, I ran them through the LogFileReader program attached to DERBY-5195 and I looked for log records involving AllocPageOperation actions. 5234_alloc.out is the result of running the following script against the transaction logs:

          #! /bin/bash

          for file in `ls -1 -r -t *.dat`
          do
          echo XXXXXXXXXXXXXXX $file XXXXXXXXXXXXXXX

          java LogFileReader $file -v | grep -A 2 -i AllocPageOperation
          done

          5234_page_10219.out was produced by running the following command to find references to actions on the bad page:

          #! /bin/bash

          for file in `ls -1 -r -t *.dat`
          do
          echo XXXXXXXXXXXXXXX $file XXXXXXXXXXXXXXX

          java LogFileReader $file -v | grep -A 2 -B 8 -i 10219
          done

          5234_summary.out is a summary of all activity on the bad page which I could find in 5234_page_10219.out.

          I don't see anything odd in the transaction logs so far, but I probably don't know what to look for. Other suggestions for what I should look for in the transaction logs?

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Attaching 5234_alloc.out, 5234_page_10219.out, and 5234_summary.out. I turned on log archiving by calling SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE_NOWAIT. This resulted in accumulating 191 transaction log files. Fortunately, this did not affect the bug. The bug continues to occur on exactly the same page at exactly the same point in the test. To examine the transaction logs, I ran them through the LogFileReader program attached to DERBY-5195 and I looked for log records involving AllocPageOperation actions. 5234_alloc.out is the result of running the following script against the transaction logs: #! /bin/bash for file in `ls -1 -r -t *.dat` do echo XXXXXXXXXXXXXXX $file XXXXXXXXXXXXXXX java LogFileReader $file -v | grep -A 2 -i AllocPageOperation done 5234_page_10219.out was produced by running the following command to find references to actions on the bad page: #! /bin/bash for file in `ls -1 -r -t *.dat` do echo XXXXXXXXXXXXXXX $file XXXXXXXXXXXXXXX java LogFileReader $file -v | grep -A 2 -B 8 -i 10219 done 5234_summary.out is a summary of all activity on the bad page which I could find in 5234_page_10219.out. I don't see anything odd in the transaction logs so far, but I probably don't know what to look for. Other suggestions for what I should look for in the transaction logs? Thanks, -Rick
          Hide
          Stefan Sitte added a comment -

          @Mike: How can i set the flags for transaction logging ?

          Show
          Stefan Sitte added a comment - @Mike: How can i set the flags for transaction logging ?
          Hide
          Mike Matrigali added a comment -

          What I would do if looking at this would be to see if you can get it to reproduce with flags set to keep all the transaction logs around. Using that with identifying when the page goes bad should get closer to where the problem is. I would look for correlation of when
          page alloc or page dealloc happens and when log records happen for the bad page.

          The stack printed has the feel of maybe looking for a page that is past the end of file? Putting some debugging about the page table and the end of file at that stack might help.

          Show
          Mike Matrigali added a comment - What I would do if looking at this would be to see if you can get it to reproduce with flags set to keep all the transaction logs around. Using that with identifying when the page goes bad should get closer to where the problem is. I would look for correlation of when page alloc or page dealloc happens and when log records happen for the bad page. The stack printed has the feel of maybe looking for a page that is past the end of file? Putting some debugging about the page table and the end of file at that stack might help.
          Hide
          Rick Hillegas added a comment -

          I also can reproduce this problem by running the repro which Stefan supplied. I will look into this bug.

          Hi Stefan: From the stack trace it appears that a base table has become corrupted. I don't know of any way to rebound from this other than to restore your database from the last good backup. Hopefully we will have a fix for this problem in 10.9, but that will only prevent the problem from happening again. The fix will not repair an already broken database.

          Show
          Rick Hillegas added a comment - I also can reproduce this problem by running the repro which Stefan supplied. I will look into this bug. Hi Stefan: From the stack trace it appears that a base table has become corrupted. I don't know of any way to rebound from this other than to restore your database from the last good backup. Hopefully we will have a fix for this problem in 10.9, but that will only prevent the problem from happening again. The fix will not repair an already broken database.
          Hide
          Stefan Sitte added a comment -

          Is there any fix or workaround available ?

          It seems to be a problem of the cache. How can i rebuild the cache after the exception occurs ?

          Show
          Stefan Sitte added a comment - Is there any fix or workaround available ? It seems to be a problem of the cache. How can i rebuild the cache after the exception occurs ?
          Hide
          Kathey Marsden added a comment -

          Triage for 10.9. Marking Critical priority, Urgent Urgency, High Value Fix. Changing component to store. I was able to reproduce with the attached repro at 10.9.0.0 alpha - (1242867)

          java.sql.SQLException: Page Page(10219,Container(0, 1168)) could not be read from disk.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142)
          at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:278)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:431)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2340)
          at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:82)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1334)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1715)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:311)
          at DbCompressErrorTester.insertData(DbCompressErrorTester.java:162)
          at DbCompressErrorTester.test(DbCompressErrorTester.java:116)
          at DbCompressErrorTester.main(DbCompressErrorTester.java:38)
          Caused by: java.sql.SQLException: Page Page(10219,Container(0, 1168)) could not be read from disk.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:42)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:12
          2)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71)
          ... 12 more
          Caused by: java.sql.SQLException: Java exception: 'Reached end of file while attempting to read a whole page.: java.io.E
          OFException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:42)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:12
          2)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142)
          at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436)
          ... 10 more
          Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:1169)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:430)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:370)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:263)
          at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:673)
          at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:193)
          at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295)
          at org.apache.derby.impl.store.raw.data.FileContainer.initPage(FileContainer.java:2430)
          at org.apache.derby.impl.store.raw.data.FileContainer.newPage(FileContainer.java:1878)
          at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(BaseContainer.java:314)
          at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(BaseContainerHandle.java:183)
          at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:302)
          at org.apache.derby.impl.store.access.heap.HeapController.insert(HeapController.java:575)
          at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:457)
          at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:999)
          at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:519)
          at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:443)
          at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:324)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1242)
          ... 5 more

          I have heard other anecdotal reports of possible corruption after introducing compress. I hope this is it.

          Show
          Kathey Marsden added a comment - Triage for 10.9. Marking Critical priority, Urgent Urgency, High Value Fix. Changing component to store. I was able to reproduce with the attached repro at 10.9.0.0 alpha - (1242867) java.sql.SQLException: Page Page(10219,Container(0, 1168)) could not be read from disk. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142) at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:278) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:431) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2340) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:82) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1334) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1715) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:311) at DbCompressErrorTester.insertData(DbCompressErrorTester.java:162) at DbCompressErrorTester.test(DbCompressErrorTester.java:116) at DbCompressErrorTester.main(DbCompressErrorTester.java:38) Caused by: java.sql.SQLException: Page Page(10219,Container(0, 1168)) could not be read from disk. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:42) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:12 2) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71) ... 12 more Caused by: java.sql.SQLException: Java exception: 'Reached end of file while attempting to read a whole page.: java.io.E OFException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:42) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:12 2) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142) at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436) ... 10 more Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page. at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:1169) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:430) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:370) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:263) at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:673) at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:193) at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295) at org.apache.derby.impl.store.raw.data.FileContainer.initPage(FileContainer.java:2430) at org.apache.derby.impl.store.raw.data.FileContainer.newPage(FileContainer.java:1878) at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(BaseContainer.java:314) at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(BaseContainerHandle.java:183) at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:302) at org.apache.derby.impl.store.access.heap.HeapController.insert(HeapController.java:575) at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:457) at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:999) at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:519) at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:443) at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:324) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1242) ... 5 more I have heard other anecdotal reports of possible corruption after introducing compress. I hope this is it.
          Hide
          Rick Hillegas added a comment -

          Re-opening this issue. I don't understand why it was closed right after an easy repro was posted.

          Show
          Rick Hillegas added a comment - Re-opening this issue. I don't understand why it was closed right after an easy repro was posted.
          Hide
          Stefan Sitte added a comment -

          That would be nice. I was suprised too, it could be reproduced in such an easy way. I hope it is a small thing to fix it.

          Show
          Stefan Sitte added a comment - That would be nice. I was suprised too, it could be reproduced in such an easy way. I hope it is a small thing to fix it.
          Hide
          Dag H. Wanvik added a comment - - edited

          Thanks for the enclosed repro, Stefan! It will increase your chances of getting a fix significantly
          I managed to reproduced the problem on Derby trunk. In my case it took about a minute.

          Show
          Dag H. Wanvik added a comment - - edited Thanks for the enclosed repro, Stefan! It will increase your chances of getting a fix significantly I managed to reproduced the problem on Derby trunk. In my case it took about a minute.
          Hide
          Stefan Sitte added a comment - - edited

          I have also problems with the inplace compress function of derby.
          With the attached snippet (DbCompressErrorTester.java) i can reproduce the "Reached end of file while attempting to read a whole page." exception. Perhaps it can help to fix the error.
          Tested with derby version 10.8.1.2 and 10.8.2.2.

          I insert some data into a table. Then i delete all data and call SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE with PURGE_ROWS=0, DEFRAGMENT_ROWS=0 and TRUNCATE_END=1. After some additional insert derby throws following exception:
          java.sql.SQLException: Page Page(10219,Container(0, 1136)) could not be read from disk.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
          at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
          at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source)
          at DbCompressErrorTester.insertData(DbCompressErrorTester.java:164)
          at DbCompressErrorTester.test(DbCompressErrorTester.java:118)
          at DbCompressErrorTester.main(DbCompressErrorTester.java:40)
          Caused by: java.sql.SQLException: Page Page(10219,Container(0, 1136)) could not be read from disk.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
          ... 13 more
          Caused by: java.sql.SQLException: Java exception: 'Reached end of file while attempting to read a whole page.: java.io.EOFException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
          at org.apache.derby.impl.jdbc.Util.javaException(Unknown Source)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
          ... 10 more
          Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(Unknown Source)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(Unknown Source)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown Source)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown Source)
          at org.apache.derby.impl.store.raw.data.CachedPage.readPage(Unknown Source)
          at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(Unknown Source)
          at org.apache.derby.impl.services.cache.ConcurrentCache.find(Unknown Source)
          at org.apache.derby.impl.store.raw.data.FileContainer.initPage(Unknown Source)
          at org.apache.derby.impl.store.raw.data.FileContainer.newPage(Unknown Source)
          at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(Unknown Source)
          at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(Unknown Source)
          at org.apache.derby.impl.store.access.heap.HeapController.doInsert(Unknown Source)
          at org.apache.derby.impl.store.access.heap.HeapController.insert(Unknown Source)
          at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(Unknown Source)
          at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(Unknown Source)
          at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source)
          at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
          at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
          ... 6 more

          Show
          Stefan Sitte added a comment - - edited I have also problems with the inplace compress function of derby. With the attached snippet (DbCompressErrorTester.java) i can reproduce the "Reached end of file while attempting to read a whole page." exception. Perhaps it can help to fix the error. Tested with derby version 10.8.1.2 and 10.8.2.2. I insert some data into a table. Then i delete all data and call SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE with PURGE_ROWS=0, DEFRAGMENT_ROWS=0 and TRUNCATE_END=1. After some additional insert derby throws following exception: java.sql.SQLException: Page Page(10219,Container(0, 1136)) could not be read from disk. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source) at DbCompressErrorTester.insertData(DbCompressErrorTester.java:164) at DbCompressErrorTester.test(DbCompressErrorTester.java:118) at DbCompressErrorTester.main(DbCompressErrorTester.java:40) Caused by: java.sql.SQLException: Page Page(10219,Container(0, 1136)) could not be read from disk. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) ... 13 more Caused by: java.sql.SQLException: Java exception: 'Reached end of file while attempting to read a whole page.: java.io.EOFException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.javaException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) ... 10 more Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page. at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(Unknown Source) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(Unknown Source) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown Source) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown Source) at org.apache.derby.impl.store.raw.data.CachedPage.readPage(Unknown Source) at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(Unknown Source) at org.apache.derby.impl.services.cache.ConcurrentCache.find(Unknown Source) at org.apache.derby.impl.store.raw.data.FileContainer.initPage(Unknown Source) at org.apache.derby.impl.store.raw.data.FileContainer.newPage(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(Unknown Source) at org.apache.derby.impl.store.access.heap.HeapController.doInsert(Unknown Source) at org.apache.derby.impl.store.access.heap.HeapController.insert(Unknown Source) at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(Unknown Source) at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(Unknown Source) at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source) at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source) at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source) ... 6 more
          Hide
          Varma R added a comment -

          @Dag - Inplace Compress command is called once every day.

          Show
          Varma R added a comment - @Dag - Inplace Compress command is called once every day.
          Hide
          Mike Matrigali added a comment -

          it is interesting that the case is exactly at a new extent which may or may not be the issue. It you have any other examples of this problem it would help to know if the end of the file looks the same as this one. Again once this problem happens a number of things
          are likely to break, and the real problem is not the insert but what allowed the allocate of the free pages to finish without the space on
          disk being forced.

          Show
          Mike Matrigali added a comment - it is interesting that the case is exactly at a new extent which may or may not be the issue. It you have any other examples of this problem it would help to know if the end of the file looks the same as this one. Again once this problem happens a number of things are likely to break, and the real problem is not the insert but what allowed the allocate of the free pages to finish without the space on disk being forced.
          Hide
          Mike Matrigali added a comment -

          I am not surprised that SYSCS_UTIL.SYSCS_COMPRESS_TABLE fixed the problem, as it seems the issue is in just the last few pages of the file that are marked free but not actually on disk. This version of compress which I refer to as offline, just reads through the used pages of the file and inserts into the new table and never looks at the problem empty and not on disk pages at the end of the file.

          Show
          Mike Matrigali added a comment - I am not surprised that SYSCS_UTIL.SYSCS_COMPRESS_TABLE fixed the problem, as it seems the issue is in just the last few pages of the file that are marked free but not actually on disk. This version of compress which I refer to as offline, just reads through the used pages of the file and inserts into the new table and never looks at the problem empty and not on disk pages at the end of the file.
          Hide
          Mike Matrigali added a comment - - edited

          The issues here all have the feel of the database getting broken sometime in past. Usually once broken not much can be
          done, including upgrading - other than rebuild. I would suggest running the consistency checker across the whole database
          to see if there are other hidden problems lurking.

          Given the end of file issues it would be interesting to know:
          o I see you run online compress once a day, maybe there is a bug in this code. Can you tell us exactly how you run this, as it has
          3 options that can be set. This code definitely executes a path that manipulates the end of file which seems to where the problem
          is. But the case you are noting more has the feel a problem in a normal allocation. Derby will extend the file the file by a group
          of pages and then force these empty new pages to disk. For some reason this group has not made it to disk, the extent map is
          indicating that it should be there, so an error results because we can't find it there. Even though it is a new page, sometimes
          a free page is really a reused page and we need some information from the old page to properly create the new page.
          o I am not too familar with HP unix, so not sure if these questions are applicable. Can you describe the file system that derby
          is running on there. Is there any chance that I/O system forces through java are not being respected by the filesystem/hardware.
          On other unix systems there are options available to force wait on synchronous writes, and derby depends on this. Unfortunately
          often the default is to pretend that write sync is happening when it is not and this can cause problems for Derby.

          Without the actual db it is hard to know what is going on exactly from just the stack. I thought derby was meant to be able to handle
          some "missing" pages at the end of the file, so this would be one case where it might be useful to try a more recent version of the
          Derby software. It could be that I am remembering recovery logic rather than runtime logic.
          The missing pages feels like a lost "force" of the free pages thus the questions about filesystem/hardware settings. Running the
          consistency checker could find other lsot forces that may have existed.

          This kind of error can also come from users manipulating files in the log directory. Have you ever "fixed" a problem where the
          database would not boot by removing files from the log directory?

          In trying to understand what might have happened to cause the bug, can you correlate at all the problems happening with machine
          crashes at all?

          Show
          Mike Matrigali added a comment - - edited The issues here all have the feel of the database getting broken sometime in past. Usually once broken not much can be done, including upgrading - other than rebuild. I would suggest running the consistency checker across the whole database to see if there are other hidden problems lurking. Given the end of file issues it would be interesting to know: o I see you run online compress once a day, maybe there is a bug in this code. Can you tell us exactly how you run this, as it has 3 options that can be set. This code definitely executes a path that manipulates the end of file which seems to where the problem is. But the case you are noting more has the feel a problem in a normal allocation. Derby will extend the file the file by a group of pages and then force these empty new pages to disk. For some reason this group has not made it to disk, the extent map is indicating that it should be there, so an error results because we can't find it there. Even though it is a new page, sometimes a free page is really a reused page and we need some information from the old page to properly create the new page. o I am not too familar with HP unix, so not sure if these questions are applicable. Can you describe the file system that derby is running on there. Is there any chance that I/O system forces through java are not being respected by the filesystem/hardware. On other unix systems there are options available to force wait on synchronous writes, and derby depends on this. Unfortunately often the default is to pretend that write sync is happening when it is not and this can cause problems for Derby. Without the actual db it is hard to know what is going on exactly from just the stack. I thought derby was meant to be able to handle some "missing" pages at the end of the file, so this would be one case where it might be useful to try a more recent version of the Derby software. It could be that I am remembering recovery logic rather than runtime logic. The missing pages feels like a lost "force" of the free pages thus the questions about filesystem/hardware settings. Running the consistency checker could find other lsot forces that may have existed. This kind of error can also come from users manipulating files in the log directory. Have you ever "fixed" a problem where the database would not boot by removing files from the log directory? In trying to understand what might have happened to cause the bug, can you correlate at all the problems happening with machine crashes at all?
          Hide
          Rick Hillegas added a comment -

          Hi, Dag. By default, the DataFileReader tool reads until it reaches EOF. You can also tell it to stop reading after a fixed number of pages.

          The page size of c450.dat is 4096, the file is 212660224 bytes long according to Varma's comment on 2011-05-18, so the number of pages in the file is 212660224 / 4096 = 51919. This agrees with what the tool found: the tool successfully decoded pages 0 - 51918.

          The extent lengths are:

          grep extentLength Output_c450.dat.xml
          <extentLength>10216</extentLength>
          <extentLength>10424</extentLength>
          <extentLength>10424</extentLength>
          <extentLength>10424</extentLength>
          <extentLength>10424</extentLength>
          <extentLength>24</extentLength>

          It does seem odd that the last allocation page thinks there are 24 more pages but there is really only 1 more page.

          Show
          Rick Hillegas added a comment - Hi, Dag. By default, the DataFileReader tool reads until it reaches EOF. You can also tell it to stop reading after a fixed number of pages. The page size of c450.dat is 4096, the file is 212660224 bytes long according to Varma's comment on 2011-05-18, so the number of pages in the file is 212660224 / 4096 = 51919. This agrees with what the tool found: the tool successfully decoded pages 0 - 51918. The extent lengths are: grep extentLength Output_c450.dat.xml <extentLength>10216</extentLength> <extentLength>10424</extentLength> <extentLength>10424</extentLength> <extentLength>10424</extentLength> <extentLength>10424</extentLength> <extentLength>24</extentLength> It does seem odd that the last allocation page thinks there are 24 more pages but there is really only 1 more page.
          Hide
          Dag H. Wanvik added a comment -

          The dump of the data pages seem to indicate that the file has no pages from page 51919 onwards (the last dumped is 515918 on c450.dat). The data dumped show that page 51917 has an extent that coincides with the one we saw dumped:
          Start/end:: 51918:62341. And page 51918 is indeed present and valid in the dump. But no further pages are printed. Rick, does the tool continue printiing till it reaches EOF or does it stop according some variable in the container stating how many pages are allocated? I am not very familiar with this code at all, but doesn't the presence of the extent information on page 51917 indicated that pages from 51918:62341 should be allocated (e.g be not handled back to the operating system after an inline compress with truncate end option[1])?

          [1] http://db.apache.org/derby/docs/10.5/ref/rrefproceduresinplacecompress.html

          Btw, Varma, does the app ever perform in-place compress?

          Show
          Dag H. Wanvik added a comment - The dump of the data pages seem to indicate that the file has no pages from page 51919 onwards (the last dumped is 515918 on c450.dat). The data dumped show that page 51917 has an extent that coincides with the one we saw dumped: Start/end:: 51918:62341. And page 51918 is indeed present and valid in the dump. But no further pages are printed. Rick, does the tool continue printiing till it reaches EOF or does it stop according some variable in the container stating how many pages are allocated? I am not very familiar with this code at all, but doesn't the presence of the extent information on page 51917 indicated that pages from 51918:62341 should be allocated (e.g be not handled back to the operating system after an inline compress with truncate end option [1] )? [1] http://db.apache.org/derby/docs/10.5/ref/rrefproceduresinplacecompress.html Btw, Varma, does the app ever perform in-place compress?
          Hide
          Rick Hillegas added a comment -

          Hi Varma,

          It's hard to say if an upgrade will fix this problem since we don't understand the root cause yet. The reason I asked whether your application experiences interrupts is that they could potentially result in a data corruption on version 10.5.3.0. Substantial work was put into protecting Derby from interrupts in 10.8.1.2. Can you say whether your application might be experiencing thread interrupts?

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Varma, It's hard to say if an upgrade will fix this problem since we don't understand the root cause yet. The reason I asked whether your application experiences interrupts is that they could potentially result in a data corruption on version 10.5.3.0. Substantial work was put into protecting Derby from interrupts in 10.8.1.2. Can you say whether your application might be experiencing thread interrupts? Thanks, -Rick
          Hide
          Varma R added a comment -

          Current Version of Derby been used in production is 10.5.3.0. Will the problem be resolved by upgrading derby database to latest version(which is 10.8.1.2)?

          Show
          Varma R added a comment - Current Version of Derby been used in production is 10.5.3.0. Will the problem be resolved by upgrading derby database to latest version(which is 10.8.1.2)?
          Hide
          Varma R added a comment - - edited

          Attached(DataFileReader_Output.zip) the XML files generated when DataFileReader.class run on KPI.KPI_MERGEIN table .DAT files.

          Show
          Varma R added a comment - - edited Attached(DataFileReader_Output.zip) the XML files generated when DataFileReader.class run on KPI.KPI_MERGEIN table .DAT files.
          Hide
          Rick Hillegas added a comment -

          Hi Varma,

          This may be a red herring, but it is worth asking: does your application experience thread interrupts?

          Some useful information might surface if you run the DataFileReader tool against the corrupt heap conglomerate. The tool is attached to DERBY-5201. Redirect the output to a file and then attach that file to this JIRA issue. Help on the tool's usage can be obtained by typing

          java DataFileReader

          I think that the following experiment would be useful (where $dbName is the name of your database directory):

          java DataFileReader $dbName/seg0/c450.dat -v -p 51916 -n 6 > c450_partial.xml

          You can also use the -d option of the tool if you are willing to disclose the data which appears in the closing pages of the file. See the help output for a description of how to use the -d option. You can, of course, redact any sensitive information in the output.

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Varma, This may be a red herring, but it is worth asking: does your application experience thread interrupts? Some useful information might surface if you run the DataFileReader tool against the corrupt heap conglomerate. The tool is attached to DERBY-5201 . Redirect the output to a file and then attach that file to this JIRA issue. Help on the tool's usage can be obtained by typing java DataFileReader I think that the following experiment would be useful (where $dbName is the name of your database directory): java DataFileReader $dbName/seg0/c450.dat -v -p 51916 -n 6 > c450_partial.xml You can also use the -d option of the tool if you are willing to disclose the data which appears in the closing pages of the file. See the help output for a description of how to use the -d option. You can, of course, redact any sensitive information in the output. Thanks, -Rick
          Hide
          Varma R added a comment -

          @Bryan, I doubt.

          The corrupted DB was moved from Production (HP-UX) to test environment (Solaris & Windows) and the DB was started with the exact settings. All three environments return the same error on INSERT in the KPI.KPI_MERGE_IN table

          Show
          Varma R added a comment - @Bryan, I doubt. The corrupted DB was moved from Production (HP-UX) to test environment (Solaris & Windows) and the DB was started with the exact settings. All three environments return the same error on INSERT in the KPI.KPI_MERGE_IN table
          Hide
          Bryan Pendleton added a comment -

          Any chance you're hitting a filesystem quota of some sort? Have you checked all the ulimit-like settings? This feels like the sort of problem where the operating system is truncating the file or partially writing it because the operating system thinks the process has hit its limit.

          Show
          Bryan Pendleton added a comment - Any chance you're hitting a filesystem quota of some sort? Have you checked all the ulimit-like settings? This feels like the sort of problem where the operating system is truncating the file or partially writing it because the operating system thinks the process has hit its limit.
          Hide
          Varma R added a comment -

          @Kristian, Thanks for the help. The data however can't be made public

          1) Ran the corrupt db data with debug libraries. Here is the output in derby.log

          ERROR XSDG0: Page Page(51919,Container(0, 1104)) could not be read from disk.
          at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:336)
          at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:695)
          at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
          at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295)
          at org.apache.derby.impl.store.raw.data.FileContainer.initPage(FileContainer.java:2317)
          at org.apache.derby.impl.store.raw.data.FileContainer.newPage(FileContainer.java:1800)
          at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(BaseContainer.java:314)
          at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(BaseContainerHandle.java:183)
          at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:302)
          at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599)
          at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452)
          at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022)
          at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495)
          at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416)
          at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235)
          at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:625)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(EmbedStatement.java:175)
          at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLIMM(DRDAConnThread.java:5022)
          at org.apache.derby.impl.drda.DRDAConnThread.processCommands(DRDAConnThread.java:750)
          at org.apache.derby.impl.drda.DRDAConnThread.run(DRDAConnThread.java:290)
          Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:474)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:242)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:212)
          at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:671)
          ... 19 more
          ============= begin nested exception, level (1) ===========
          java.io.EOFException: Reached end of file while attempting to read a whole page.
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:474)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:242)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:212)
          at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:671)
          at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
          at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295)
          at org.apache.derby.impl.store.raw.data.FileContainer.initPage(FileContainer.java:2317)
          at org.apache.derby.impl.store.raw.data.FileContainer.newPage(FileContainer.java:1800)
          at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(BaseContainer.java:314)
          at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(BaseContainerHandle.java:183)
          at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:302)
          at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599)
          at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452)
          at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022)
          at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495)
          at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416)
          at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235)
          at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:625)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(EmbedStatement.java:175)
          at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLIMM(DRDAConnThread.java:5022)
          at org.apache.derby.impl.drda.DRDAConnThread.processCommands(DRDAConnThread.java:750)
          at org.apache.derby.impl.drda.DRDAConnThread.run(DRDAConnThread.java:290)
          ============= end nested exception, level (1) ===========

          ------------ END SHUTDOWN ERROR STACK -------------

          DEBUG FileContainer OUTPUT: got exception from initPage:
          reuse = true
          syncFlag = 0
          allocPage = *** Alloc page ***
          nextAllocPageNumber = -1
          nextAllocPageOffset = 0
          reserved1 = 0
          reserved2 = 0
          reserved3 = 0
          reserved4 = 0
          borrowedSpace = 0
          extent = ------------------------------------------------------------------------------
          Extent map of from page 51918 to page 62341
          page 51918: valid, in use page
          page 51919: free page
          page 51920: free page
          page 51921: free page
          page 51922: free page
          page 51923: free page
          page 51924: free page
          page 51925: free page
          page 51926: free page
          page 51927: free page
          page 51928: free page
          page 51929: free page
          page 51930: free page
          page 51931: free page
          page 51932: free page
          page 51933: free page
          page 51934: free page
          page 51935: free page
          page 51936: free page
          page 51937: free page
          page 51938: free page
          page 51939: free page
          page 51940: free page
          page 51941: free page
          From 51941 to 62341 are un-allocated pages
          ------------------------------------------------------------------------------

          ---------------------------------------------------
          page id: Page(51917,Container(0, 1104)) Overflow: false PageVersion: 51 SlotsInUse: 0 DeletedRowCount: 0 PageStatus: 1 NextId: 6 firstFreeByte: 109 freeSpace: 3979 totalSpace: 3979 spareSpace: 20% minimumRecordSize : 12 PageSize: 4096
          ---------------------------------------------------
          Hex dump:
          00000000: 0076 0000 0001 0000 0000 0000 0033 0000 .v...........3..
          00000010: 0000 0006 0000 0000 0000 0000 0000 0000 ................
          00000020: 0000 0000 0001 0000 0000 0000 0000 0000 ................
          00000030: 0000 0000 0000 0000 0000 0000 ffff ffff ................
          00000040: ffff ffff 0000 0000 0000 0000 0000 0000 ................
          00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000070: 000c ace0 0000 0000 0000 00ca ce00 0000 ..?........??...
          00000080: 0000 00f3 8500 0000 1830 0000 1200 0000 ....?....0......
          00000090: 1800 0000 0000 0000 0000 0000 0000 0000 ................
          000000a0: 0000 0000 0000 0000 187f ffff 0000 0018 ................
          000000b0: 7fff ff00 0000 0000 0000 0000 0000 0000 ................
          000000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000000e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000000f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000100: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000110: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000120: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000130: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000140: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000150: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000160: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000170: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000180: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000190: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000001a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000001b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000001c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000001f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000200: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000210: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000220: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000230: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000240: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000250: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000260: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000270: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000280: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000290: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000002a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000002b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000002c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000002d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000002e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000002f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000300: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000310: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000320: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000330: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000340: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000350: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000360: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000370: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000380: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000390: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000003a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000003b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000003c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000003d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000003e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000003f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000400: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000410: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000420: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000430: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000440: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000450: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000460: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000470: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000480: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000490: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000004a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000004b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000004c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000004d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000004e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000004f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000500: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000510: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000520: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000530: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000540: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000550: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000560: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000570: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000580: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000590: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000005a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000005b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000005c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000005d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000005e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000005f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000600: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000610: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000620: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000630: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000640: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000650: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000660: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000670: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000680: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000690: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000006a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000006b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000006c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000006d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000006e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000006f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000700: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000710: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000720: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000730: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000740: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000750: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000760: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000770: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000780: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000790: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000007a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000007b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000007c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000007d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000007e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000007f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000800: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000810: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000820: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000830: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000840: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000850: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000860: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000870: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000880: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000890: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000008a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000008b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000008c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000008d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000008e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000008f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000900: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000910: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000920: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000930: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000940: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000950: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000960: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000970: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000980: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000990: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000009a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000009b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000009c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000009d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000009e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000009f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000a00: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000a10: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000a20: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000a30: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000a40: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000a50: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000a60: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000a70: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000a80: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000a90: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000aa0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000ab0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000ac0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000ad0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000ae0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000af0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000b00: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000b10: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000b20: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000b30: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000b40: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000b50: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000b60: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000b70: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000b80: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000b90: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000ba0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000bb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000bc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000bd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000be0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000bf0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000c00: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000c10: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000c20: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000c30: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000c40: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000c50: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000c60: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000c70: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000c80: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000c90: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000ca0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000cb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000cc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000cd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000ce0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000cf0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000d00: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000d10: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000d20: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000d30: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000d40: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000d50: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000d60: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000d70: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000d80: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000d90: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000da0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000db0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000dc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000dd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000de0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000df0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000e00: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000e10: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000e20: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000e30: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000e40: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000e50: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000e60: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000e70: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000e80: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000e90: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000ea0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000eb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000ec0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000ed0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000ee0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000ef0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000f00: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000f10: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000f20: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000f30: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000f40: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000f50: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000f60: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000f70: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000f80: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000f90: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000fa0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000fb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000fc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000fd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000fe0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000ff0: 0000 0000 0000 0000 0000 0000 4dc3 fccd ............M?.?
          ---------------------------------------------------

          2011-05-18 12:30:24.189 GMT Thread[DRDAConnThread_3,5,main] (XID = 28699230), (SESSIONID = 1), (DATABASE = kpidb), (DRDAID = GAEDD120.O4F6-4110096304207220172

          {1}), Cleanup action starting
          2011-05-18 12:30:24.189 GMT Thread[DRDAConnThread_3,5,main] (XID = 28699230), (SESSIONID = 1), (DATABASE = kpidb), (DRDAID = GAEDD120.O4F6-4110096304207220172{1}

          ), Failed Statement is: INSERT INTO KPI.KPI_MERGEIN (A0_TXN_ID, A1_NE_ID, A2_CHU_IP_ADDR, A3_BATCH_DATE,A5_CODE) VALUES (-1, 'BMTDE', '192.2.1.3', 231456879, 'KSD')
          ERROR XSDG0: Page Page(51919,Container(0, 1104)) could not be read from disk.
          at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:336)
          at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:695)
          at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
          at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295)
          at org.apache.derby.impl.store.raw.data.FileContainer.initPage(FileContainer.java:2317)
          at org.apache.derby.impl.store.raw.data.FileContainer.newPage(FileContainer.java:1800)
          at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(BaseContainer.java:314)
          at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(BaseContainerHandle.java:183)
          at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:302)
          at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599)
          at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452)
          at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022)
          at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495)
          at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416)
          at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235)
          at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:625)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(EmbedStatement.java:175)
          at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLIMM(DRDAConnThread.java:5022)
          at org.apache.derby.impl.drda.DRDAConnThread.processCommands(DRDAConnThread.java:750)
          at org.apache.derby.impl.drda.DRDAConnThread.run(DRDAConnThread.java:290)
          Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:474)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:242)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:212)
          at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:671)
          ... 19 more
          ============= begin nested exception, level (1) ===========
          java.io.EOFException: Reached end of file while attempting to read a whole page.
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:474)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:242)
          at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:212)
          at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:671)
          at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
          at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295)
          at org.apache.derby.impl.store.raw.data.FileContainer.initPage(FileContainer.java:2317)
          at org.apache.derby.impl.store.raw.data.FileContainer.newPage(FileContainer.java:1800)
          at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(BaseContainer.java:314)
          at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(BaseContainerHandle.java:183)
          at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:302)
          at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599)
          at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452)
          at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022)
          at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495)
          at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416)
          at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235)
          at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:625)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(EmbedStatement.java:175)
          at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLIMM(DRDAConnThread.java:5022)
          at org.apache.derby.impl.drda.DRDAConnThread.processCommands(DRDAConnThread.java:750)
          at org.apache.derby.impl.drda.DRDAConnThread.run(DRDAConnThread.java:290)
          ============= end nested exception, level (1) ===========

          2011-05-18 12:30:24.189 GMT:
          Shutting down instance a816c00e-0130-0313-18c9-ffff874c3ff6
          ----------------------------------------------------------------
          Cleanup action completed

          2) The production environment does include delete scripts to delete data older than 7 days and it is run every 15 minutes. Also SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE procedure is run once every day to release free pages to OS.

          Intersting observation however is, it is SYSCS_UTIL.SYSCS_COMPRESS_TABLE procedure on the corrupt db that made it work. Running SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE procedure on the corrupt db returned the following error.

          ERROR 38000: DERBY SQL error: SQLCODE: -1, SQLSTATE: 38000, SQLERRMC: java.sql.SQLException: Page Page(51919,Container(0, 1104)) could not be read from disk.38000XSDG0:Page(51919,Container(0, 1104))4096XSDG0.DXJ001:java.io.EOFExceptionReached end of file while attempting to read a whole page.XJ001.U
          ERROR XSDG0: DERBY SQL error: SQLCODE: 0, SQLSTATE: XSDG0, SQLERRMC: Page(51919,Container(0, 1104))4096XSDG0.D
          ERROR XJ001: DERBY SQL error: SQLCODE: 0, SQLSTATE: XJ001, SQLERRMC: java.io.EOFExceptionReached end of file while attempting to read a whole page.XJ001.U

          Would it be recommeneded to run the other compress script instead of inplace compress script? Also would upgrading to newer version would help.

          @Rick , regarding Page size query - the file system are indeed mutiple of 32Kb

          KPI_IN
          254169088 c400.dat
          44179456 c501.dat
          48295936 c581.dat
          197808128 c481.dat
          KPI_ERROR1
          8192 c591.dat
          8192 c511.dat
          8192 c491.dat
          8192 c410.dat
          KPI_ERROR2
          1630208 c420.dat
          258048 c5a1.dat
          184320 c521.dat
          2142208 c4a1.dat
          KPI_ERRORMERGE
          8192 c4b1.dat
          8192 c430.dat
          8192 c5b1.dat
          8192 c531.dat
          KPI_DROPPED
          4980736 c5c1.dat
          3477504 c541.dat
          15597568 c4c1.dat
          31903744 c440.dat
          KPI_MERGEIN
          43851776 c5d1.dat
          27942912 c551.dat
          148070400 c4d1.dat
          212660224 c450.dat
          KPI_MERGEOUT
          121430016 c4e1.dat
          48001024 c460.dat
          45146112 c5e1.dat
          19886080 c561.dat
          KPI_OUT
          7487488 c571.dat
          15474688 c5f1.dat
          46276608 c4f1.dat
          57602048 c470.dat

          Show
          Varma R added a comment - @Kristian, Thanks for the help. The data however can't be made public 1) Ran the corrupt db data with debug libraries. Here is the output in derby.log ERROR XSDG0: Page Page(51919,Container(0, 1104)) could not be read from disk. at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:336) at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:695) at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190) at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295) at org.apache.derby.impl.store.raw.data.FileContainer.initPage(FileContainer.java:2317) at org.apache.derby.impl.store.raw.data.FileContainer.newPage(FileContainer.java:1800) at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(BaseContainer.java:314) at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(BaseContainerHandle.java:183) at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:302) at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599) at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452) at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022) at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495) at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416) at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235) at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:625) at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(EmbedStatement.java:175) at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLIMM(DRDAConnThread.java:5022) at org.apache.derby.impl.drda.DRDAConnThread.processCommands(DRDAConnThread.java:750) at org.apache.derby.impl.drda.DRDAConnThread.run(DRDAConnThread.java:290) Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page. at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:474) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:242) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:212) at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:671) ... 19 more ============= begin nested exception, level (1) =========== java.io.EOFException: Reached end of file while attempting to read a whole page. at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:474) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:242) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:212) at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:671) at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190) at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295) at org.apache.derby.impl.store.raw.data.FileContainer.initPage(FileContainer.java:2317) at org.apache.derby.impl.store.raw.data.FileContainer.newPage(FileContainer.java:1800) at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(BaseContainer.java:314) at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(BaseContainerHandle.java:183) at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:302) at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599) at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452) at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022) at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495) at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416) at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235) at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:625) at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(EmbedStatement.java:175) at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLIMM(DRDAConnThread.java:5022) at org.apache.derby.impl.drda.DRDAConnThread.processCommands(DRDAConnThread.java:750) at org.apache.derby.impl.drda.DRDAConnThread.run(DRDAConnThread.java:290) ============= end nested exception, level (1) =========== ------------ END SHUTDOWN ERROR STACK ------------- DEBUG FileContainer OUTPUT: got exception from initPage: reuse = true syncFlag = 0 allocPage = *** Alloc page *** nextAllocPageNumber = -1 nextAllocPageOffset = 0 reserved1 = 0 reserved2 = 0 reserved3 = 0 reserved4 = 0 borrowedSpace = 0 extent = ------------------------------------------------------------------------------ Extent map of from page 51918 to page 62341 page 51918: valid, in use page page 51919: free page page 51920: free page page 51921: free page page 51922: free page page 51923: free page page 51924: free page page 51925: free page page 51926: free page page 51927: free page page 51928: free page page 51929: free page page 51930: free page page 51931: free page page 51932: free page page 51933: free page page 51934: free page page 51935: free page page 51936: free page page 51937: free page page 51938: free page page 51939: free page page 51940: free page page 51941: free page From 51941 to 62341 are un-allocated pages ------------------------------------------------------------------------------ --------------------------------------------------- page id: Page(51917,Container(0, 1104)) Overflow: false PageVersion: 51 SlotsInUse: 0 DeletedRowCount: 0 PageStatus: 1 NextId: 6 firstFreeByte: 109 freeSpace: 3979 totalSpace: 3979 spareSpace: 20% minimumRecordSize : 12 PageSize: 4096 --------------------------------------------------- Hex dump: 00000000: 0076 0000 0001 0000 0000 0000 0033 0000 .v...........3.. 00000010: 0000 0006 0000 0000 0000 0000 0000 0000 ................ 00000020: 0000 0000 0001 0000 0000 0000 0000 0000 ................ 00000030: 0000 0000 0000 0000 0000 0000 ffff ffff ................ 00000040: ffff ffff 0000 0000 0000 0000 0000 0000 ................ 00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000070: 000c ace0 0000 0000 0000 00ca ce00 0000 ..?........??... 00000080: 0000 00f3 8500 0000 1830 0000 1200 0000 ....?....0...... 00000090: 1800 0000 0000 0000 0000 0000 0000 0000 ................ 000000a0: 0000 0000 0000 0000 187f ffff 0000 0018 ................ 000000b0: 7fff ff00 0000 0000 0000 0000 0000 0000 ................ 000000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000000e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000000f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000100: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000110: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000120: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000130: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000140: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000150: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000160: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000170: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000180: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000190: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000200: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000210: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000220: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000230: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000240: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000250: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000260: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000270: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000280: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000290: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000002a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000002b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000002c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000002d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000002e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000002f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000300: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000310: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000320: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000330: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000340: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000350: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000360: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000370: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000380: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000390: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000003a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000003b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000003c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000003d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000003e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000003f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000400: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000410: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000420: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000430: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000440: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000450: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000460: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000470: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000480: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000490: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000004a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000004b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000004c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000004d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000004e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000004f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000500: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000510: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000520: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000530: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000540: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000550: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000560: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000570: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000580: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000590: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000005a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000005b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000005c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000005d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000005e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000005f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000600: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000610: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000620: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000630: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000640: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000650: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000660: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000670: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000680: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000690: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000006a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000006b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000006c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000006d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000006e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000006f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000700: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000710: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000720: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000730: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000740: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000750: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000760: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000770: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000780: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000790: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000007a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000007b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000007c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000007d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000007e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000007f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000800: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000810: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000820: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000830: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000840: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000850: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000860: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000870: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000880: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000890: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000008a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000008b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000008c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000008d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000008e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000008f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000900: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000910: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000920: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000930: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000940: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000950: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000960: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000970: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000980: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000990: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000009a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000009b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000009c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000009d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000009e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000009f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000a00: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000a10: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000a20: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000a30: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000a40: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000a50: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000a60: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000a70: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000a80: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000a90: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000aa0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000ab0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000ac0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000ad0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000ae0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000af0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000b00: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000b10: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000b20: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000b30: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000b40: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000b50: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000b60: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000b70: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000b80: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000b90: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000ba0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000bb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000bc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000bd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000be0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000bf0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c00: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c10: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c20: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c30: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c40: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c50: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c60: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c70: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c80: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c90: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000ca0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000cb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000cc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000cd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000ce0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000cf0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000d00: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000d10: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000d20: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000d30: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000d40: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000d50: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000d60: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000d70: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000d80: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000d90: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000da0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000db0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000dc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000dd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000de0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000df0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000e00: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000e10: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000e20: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000e30: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000e40: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000e50: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000e60: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000e70: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000e80: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000e90: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000ea0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000eb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000ec0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000ed0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000ee0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000ef0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f00: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f10: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f20: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f30: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f40: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f50: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f60: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f70: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f80: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f90: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000fa0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000fb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000fc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000fd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000fe0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000ff0: 0000 0000 0000 0000 0000 0000 4dc3 fccd ............M?.? --------------------------------------------------- 2011-05-18 12:30:24.189 GMT Thread [DRDAConnThread_3,5,main] (XID = 28699230), (SESSIONID = 1), (DATABASE = kpidb), (DRDAID = GAEDD120.O4F6-4110096304207220172 {1}), Cleanup action starting 2011-05-18 12:30:24.189 GMT Thread [DRDAConnThread_3,5,main] (XID = 28699230), (SESSIONID = 1), (DATABASE = kpidb), (DRDAID = GAEDD120.O4F6-4110096304207220172{1} ), Failed Statement is: INSERT INTO KPI.KPI_MERGEIN (A0_TXN_ID, A1_NE_ID, A2_CHU_IP_ADDR, A3_BATCH_DATE,A5_CODE) VALUES (-1, 'BMTDE', '192.2.1.3', 231456879, 'KSD') ERROR XSDG0: Page Page(51919,Container(0, 1104)) could not be read from disk. at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:336) at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:695) at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190) at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295) at org.apache.derby.impl.store.raw.data.FileContainer.initPage(FileContainer.java:2317) at org.apache.derby.impl.store.raw.data.FileContainer.newPage(FileContainer.java:1800) at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(BaseContainer.java:314) at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(BaseContainerHandle.java:183) at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:302) at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599) at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452) at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022) at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495) at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416) at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235) at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:625) at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(EmbedStatement.java:175) at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLIMM(DRDAConnThread.java:5022) at org.apache.derby.impl.drda.DRDAConnThread.processCommands(DRDAConnThread.java:750) at org.apache.derby.impl.drda.DRDAConnThread.run(DRDAConnThread.java:290) Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page. at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:474) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:242) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:212) at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:671) ... 19 more ============= begin nested exception, level (1) =========== java.io.EOFException: Reached end of file while attempting to read a whole page. at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:474) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:242) at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:212) at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:671) at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190) at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295) at org.apache.derby.impl.store.raw.data.FileContainer.initPage(FileContainer.java:2317) at org.apache.derby.impl.store.raw.data.FileContainer.newPage(FileContainer.java:1800) at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(BaseContainer.java:314) at org.apache.derby.impl.store.raw.data.BaseContainerHandle.addPage(BaseContainerHandle.java:183) at org.apache.derby.impl.store.access.heap.HeapController.doInsert(HeapController.java:302) at org.apache.derby.impl.store.access.heap.HeapController.insertAndFetchLocation(HeapController.java:599) at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:452) at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:1022) at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:495) at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:416) at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:297) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1235) at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:625) at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(EmbedStatement.java:175) at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLIMM(DRDAConnThread.java:5022) at org.apache.derby.impl.drda.DRDAConnThread.processCommands(DRDAConnThread.java:750) at org.apache.derby.impl.drda.DRDAConnThread.run(DRDAConnThread.java:290) ============= end nested exception, level (1) =========== 2011-05-18 12:30:24.189 GMT: Shutting down instance a816c00e-0130-0313-18c9-ffff874c3ff6 ---------------------------------------------------------------- Cleanup action completed 2) The production environment does include delete scripts to delete data older than 7 days and it is run every 15 minutes. Also SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE procedure is run once every day to release free pages to OS. Intersting observation however is, it is SYSCS_UTIL.SYSCS_COMPRESS_TABLE procedure on the corrupt db that made it work. Running SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE procedure on the corrupt db returned the following error. ERROR 38000: DERBY SQL error: SQLCODE: -1, SQLSTATE: 38000, SQLERRMC: java.sql.SQLException: Page Page(51919,Container(0, 1104)) could not be read from disk.38000XSDG0:Page(51919,Container(0, 1104))4096XSDG0.DXJ001:java.io.EOFExceptionReached end of file while attempting to read a whole page.XJ001.U ERROR XSDG0: DERBY SQL error: SQLCODE: 0, SQLSTATE: XSDG0, SQLERRMC: Page(51919,Container(0, 1104))4096XSDG0.D ERROR XJ001: DERBY SQL error: SQLCODE: 0, SQLSTATE: XJ001, SQLERRMC: java.io.EOFExceptionReached end of file while attempting to read a whole page.XJ001.U Would it be recommeneded to run the other compress script instead of inplace compress script? Also would upgrading to newer version would help. @Rick , regarding Page size query - the file system are indeed mutiple of 32Kb KPI_IN 254169088 c400.dat 44179456 c501.dat 48295936 c581.dat 197808128 c481.dat KPI_ERROR1 8192 c591.dat 8192 c511.dat 8192 c491.dat 8192 c410.dat KPI_ERROR2 1630208 c420.dat 258048 c5a1.dat 184320 c521.dat 2142208 c4a1.dat KPI_ERRORMERGE 8192 c4b1.dat 8192 c430.dat 8192 c5b1.dat 8192 c531.dat KPI_DROPPED 4980736 c5c1.dat 3477504 c541.dat 15597568 c4c1.dat 31903744 c440.dat KPI_MERGEIN 43851776 c5d1.dat 27942912 c551.dat 148070400 c4d1.dat 212660224 c450.dat KPI_MERGEOUT 121430016 c4e1.dat 48001024 c460.dat 45146112 c5e1.dat 19886080 c561.dat KPI_OUT 7487488 c571.dat 15474688 c5f1.dat 46276608 c4f1.dat 57602048 c470.dat
          Hide
          Kristian Waagan added a comment -

          Varma, some comments below:

          a) It's not clear to me if you run with the Derby debug build, i.e. the debug bundle named "db-derby-10.x.y.z-lib-debug.zip" (downloadable from the Apache Derby site).
          My hope was that it would detect if there was invalid metadata when accessing the table.

          b) This is probably a dead end anyway (unless item d suggests otherwise). In any case, I think it would only matter on the production system since I assume that's where the data was actually written to disk.

          c) Interesting. So it appears that the compress operation is able to rectify the state that's causing the insert to fail.
          If asked for, could the data file for the apparently corrupted table (or the whole database) be made public?

          Compressing a table is done to reclaim unused disk space for the base table and the indexes. If your load consists of a high rate of deletes and updates, it might be worth running it periodically if possible. See the Reference Guide and "Reclaiming unused space" in the Derby Server and Administration Guide for details.

          I'm hoping someone with a more intimate knowledge about the Derby store can point us in the right direction, and it would be nice to obtain the size of the corrupted table.
          It seems Derby fails to read back a newly created page, which is odd...

          Show
          Kristian Waagan added a comment - Varma, some comments below: a) It's not clear to me if you run with the Derby debug build, i.e. the debug bundle named "db-derby-10.x.y.z-lib-debug.zip" (downloadable from the Apache Derby site). My hope was that it would detect if there was invalid metadata when accessing the table. b) This is probably a dead end anyway (unless item d suggests otherwise). In any case, I think it would only matter on the production system since I assume that's where the data was actually written to disk. c) Interesting. So it appears that the compress operation is able to rectify the state that's causing the insert to fail. If asked for, could the data file for the apparently corrupted table (or the whole database) be made public? Compressing a table is done to reclaim unused disk space for the base table and the indexes. If your load consists of a high rate of deletes and updates, it might be worth running it periodically if possible. See the Reference Guide and "Reclaiming unused space" in the Derby Server and Administration Guide for details. I'm hoping someone with a more intimate knowledge about the Derby store can point us in the right direction, and it would be nice to obtain the size of the corrupted table. It seems Derby fails to read back a newly created page, which is odd...
          Hide
          Rick Hillegas added a comment -

          Hi Varma,

          To find the data files corresponding to your tables, you can run the following script:

          create function toHexString( a bigint ) returns varchar( 25 )
          language java parameter style java no sql
          external name 'java.lang.Long.toHexString';

          select t.tableName, 'c' || toHexString( c.conglomerateNumber ) || '.dat'
          from sys.systables t, sys.sysconglomerates c, sys.sysschemas s
          where t.tableID = c.tableID
          and t.schemaID = s.schemaID
          and s.schemaName not like 'SYS%';

          drop function toHexString;

          Show
          Rick Hillegas added a comment - Hi Varma, To find the data files corresponding to your tables, you can run the following script: create function toHexString( a bigint ) returns varchar( 25 ) language java parameter style java no sql external name 'java.lang.Long.toHexString'; select t.tableName, 'c' || toHexString( c.conglomerateNumber ) || '.dat' from sys.systables t, sys.sysconglomerates c, sys.sysschemas s where t.tableID = c.tableID and t.schemaID = s.schemaID and s.schemaName not like 'SYS%'; drop function toHexString;
          Hide
          Varma R added a comment -

          Hi,

          Tried the above options mentioned.
          a) Given the below debug option which starting the derby database.
          -Dderby.debug.true=CacheTrace,DaemonTrace,DeadLockTrace,LockStackTrace,LockTrace,LogTrace,memoryLeakTrace,ScanTrace,SerializedTrace,SpaceTrace,TEST_LOG_FULL

          But this didn't print any additional information in the logs.

          b) Below are the command output's on test environment.
          -bash-3.00$ ulimit -a | grep file
          core file size (blocks, -c) unlimited
          file size (blocks, -f) unlimited
          open files (-n) 256
          -bash-3.00$ plimit 5036
          5036: java -Dderby.system.home=/export/home/cmd -Dderby.locks.monitor=true -
          resource current maximum
          time(seconds) unlimited unlimited
          file(blocks) unlimited unlimited
          data(kbytes) unlimited unlimited
          stack(kbytes) 10240 unlimited
          coredump(blocks) unlimited unlimited
          nofiles(descriptors) 65536 65536
          vmemory(kbytes) unlimited unlimited

          you can see that there are no limits on the file system and 65536 is the hard limit for the file descriptors.

          c) After running SYSCS_UTIL.SYSCS_COMPRESS_TABLE procedure on KPI.KPI_MERGEIN table. INSERT command is working on KPI.KPI_MERGEIN table.

          d) How to find out the list of .dat files corresponding to a table in seg0 directory?

          This same application was load tested for higher transcation volume and no such error were reported during development. However the problem in production seems to be pointing to cumulative data/disk usage. Also would it be recommeneded to run SYSCS_UTIL.SYSCS_COMPRESS_TABLE once in a while?

          Show
          Varma R added a comment - Hi, Tried the above options mentioned. a) Given the below debug option which starting the derby database. -Dderby.debug.true=CacheTrace,DaemonTrace,DeadLockTrace,LockStackTrace,LockTrace,LogTrace,memoryLeakTrace,ScanTrace,SerializedTrace,SpaceTrace,TEST_LOG_FULL But this didn't print any additional information in the logs. b) Below are the command output's on test environment. -bash-3.00$ ulimit -a | grep file core file size (blocks, -c) unlimited file size (blocks, -f) unlimited open files (-n) 256 -bash-3.00$ plimit 5036 5036: java -Dderby.system.home=/export/home/cmd -Dderby.locks.monitor=true - resource current maximum time(seconds) unlimited unlimited file(blocks) unlimited unlimited data(kbytes) unlimited unlimited stack(kbytes) 10240 unlimited coredump(blocks) unlimited unlimited nofiles(descriptors) 65536 65536 vmemory(kbytes) unlimited unlimited you can see that there are no limits on the file system and 65536 is the hard limit for the file descriptors. c) After running SYSCS_UTIL.SYSCS_COMPRESS_TABLE procedure on KPI.KPI_MERGEIN table. INSERT command is working on KPI.KPI_MERGEIN table. d) How to find out the list of .dat files corresponding to a table in seg0 directory? This same application was load tested for higher transcation volume and no such error were reported during development. However the problem in production seems to be pointing to cumulative data/disk usage. Also would it be recommeneded to run SYSCS_UTIL.SYSCS_COMPRESS_TABLE once in a while?
          Hide
          Kristian Waagan added a comment -

          Hi,

          I can't say what has gone wrong, but have a few initial questions that may inform the debugging process:
          a) Can you run with the debug version of Derby?
          This would be best during normal operation, before the error has happened, but may not work because all the extra asserts slow down Derby a lot.
          Booting and accessing (SELECT/INSERT) the corrupted database with the debug version is also useful in some cases, and that's a lot simpler to do.
          b) Have you verified that there are no limits on the size of the database files, i.e. OS quotas or file system limitations, in the production environment?
          c) What happens if you run SYSCS_UTIL.SYSCS_COMPRESS_TABLE on KPI.KPI_MERGEIN after the insert fails?
          If the compress-operation succeeds, does the insert still fail afterwards?
          d) Looking at the data file for the corrupted table, is the size a multiple of the page size?

          The failing code is using NIO, but there doesn't seem to be a problem with interrupts (based on current info only). The two obvious ways the code can fail are if the file for some reason has been truncated or that the position argument is incorrect.

          Show
          Kristian Waagan added a comment - Hi, I can't say what has gone wrong, but have a few initial questions that may inform the debugging process: a) Can you run with the debug version of Derby? This would be best during normal operation, before the error has happened, but may not work because all the extra asserts slow down Derby a lot. Booting and accessing (SELECT/INSERT) the corrupted database with the debug version is also useful in some cases, and that's a lot simpler to do. b) Have you verified that there are no limits on the size of the database files, i.e. OS quotas or file system limitations, in the production environment? c) What happens if you run SYSCS_UTIL.SYSCS_COMPRESS_TABLE on KPI.KPI_MERGEIN after the insert fails? If the compress-operation succeeds, does the insert still fail afterwards? d) Looking at the data file for the corrupted table, is the size a multiple of the page size? The failing code is using NIO, but there doesn't seem to be a problem with interrupts (based on current info only). The two obvious ways the code can fail are if the file for some reason has been truncated or that the position argument is incorrect.

            People

            • Assignee:
              Unassigned
              Reporter:
              Varma R
            • Votes:
              7 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development