[DERBY-4239] Possible corruption if SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE is called during checkpoint

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 10.1.3.3, 10.2.2.1, 10.3.2.1, 10.4.2.0, 10.5.1.1, 10.6.1.0
• Fix Version/s: 10.1.3.3, 10.2.2.1, 10.3.3.1, 10.4.2.1, 10.5.2.0, 10.6.1.0
• Component/s: Store

Environment

z/OS z10 processor.
java version "1.6.0"
Java(TM) SE Runtime Environment (build pmz3160sr4-20090219_01(SR4))
IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 z/OS s390-31 jvmmz3160-20090215_29883 (JIT enabled, AOT enabled)
J9VM - 20090215_029883_bHdSMr
JIT - r9_20090213_2028
GC - 20090213_AA)
JCL - 20090218_01
also
java version "1.6.0"
Java(TM) SE Runtime Environment (build pmz3160sr2ifix-20081021_01(SR2+IZ32776+IZ33456))
IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 z/OS s390-31 jvmmz3160ifx-20081010_24288 (JIT enabled, AOT enabled)
J9VM - 20081009_024288_bHdSMr
JIT - r9_20080721_1330ifx2
GC - 20080724_AA)
JCL - 20080808_02

• Bug behavior facts: Data corruption, Regression Test Failure
• Assignee: Mike Matrigali
• Reporter: Katherine Marsden
• Created: 20/May/09 23:36
• Updated: 12/Jan/11 22:24
• Resolved: 05/Jun/09 15:33

Description

corruption with storerecovery oc_rec? tests. ERROR XSLA7: Cannot redo operation null in the log when compress occurs during checkpoint, then jvm exits

I saw corruption on z/OS with the storerecovery tests and 10.5.1.1. The failure comes in oc_rec3 trying to connect to the database, but the actual problem seems to have occurred with the prior test oc_rec2. The problem is somewhat intermittent, happening approximately 1/4 times. I extracted the case from the harness and will attach the reproduction and run the script repro.ksh. The script will loop up to 50 times until it gets the failure which looks like.

ERROR XSLA7: Cannot redo operation null in the log.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.store.raw.log.FileLogger.redo(Unknown Source)
at org.apache.derby.impl.store.raw.log.LogToFile.recover(Unknown Source)
at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source)
at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)
at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:311)
at java.sql.DriverManager.getConnection(DriverManager.java:268)
at CheckTables.main(CheckTables.java:8)
Caused by: ERROR XSDBB: Unknown page format at page Page(16,Container(0, 1073)), page dump follows: Hex dump:
00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
<snip lots of 000's>

I ran it with 10.3 and it completed all 50 iterations, so whether JVM or Derby issue it seems new since 10.3. (I haven't tried with 10.4). Oddly I have run tests many times before on this machine using in the 10.5.1.1 release and the same jvm and have never seen this failure, so am looking into whether maybe something changed on the machine or environment.

Issue Links

• is duplicated by: DERBY-3393 Database corruption when adding sleep() in RAFContainer4.writePage()
• is duplicated by: DERBY-4075 ERROR XSDBB: Unknown page format at page Page(613,Container(0, 1024)) when running MailJdbc (Embedded) system tests
• is related to: DERBY-3393 Database corruption when adding sleep() in RAFContainer4.writePage()
• is related to: DERBY-3833 OnlineCompressTest fails with ERROR XSDBB: Unknown page format at page Page(1,Container(0, 1440)) on 10.3 with ibm 1.5
• relates to: DERBY-4252 intermittent corruption./java.io.EOFException in RandomAccessFile.readInt() on boot after compress with background checkpoint