Derby
  1. Derby
  2. DERBY-3607

Invalid checksum error in Derby 10.3.2.1

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Cannot Reproduce
    • Affects Version/s: 10.3.2.1, 10.4.1.3
    • Fix Version/s: None
    • Component/s: Store
    • Environment:
      OS-WIN XP SP2, 1.86GHz, 2GB, JVM 1.5, disk caching disabled, Hibernate 3.1.1.RC3,c3p0
    • Urgency:
      Normal
    • Bug behavior facts:
      Data corruption, Seen in production

      Description

      I am getting this execption when ever I try to restart my application

      java.sql.SQLException: Invalid checksum on Page Page(0,Container(0, 2033)), expected=2,731,401,932, on-disk version=2,375,776,513, page dump follows: Hex dump:

      00000000: 0076 0000 0001 0000 0000 0000 0002 0000 .v..............
      00000010: 0000 0006 0000 0000 0000 0000 0000 0000 ................
      00000020: 0000 0000 0001 0000 0000 0000 0000 0000 ................
      00000030: 0000 0000 0000 0000 0000 0000 ffff ffff ................
      00000040: ffff ffff 0000 0000 0000 0000 0000 0000 ................
      00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
      00000060: 0000 0000 0000 0000 0000 0000 5000 0000 ............P...

      at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
      at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedCallableStatement.executeStatement(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedPreparedStatement.execute(Unknown Source)
      at com.mchange.v2.c3p0.impl.NewProxyCallableStatement.execute(NewProxyCallableStatement.java:3044)
      at ae.sphere.arena.database.management.backup.BackupStategy.createBackup(BackupStategy.java:56)
      at ae.sphere.arena.database.management.backup.BackupStategy.doSchedulerJob(BackupStategy.java:41)
      at ae.sphere.arena.common.jobscheduler.Scheduler$1.run(Scheduler.java:49)
      at org.eclipse.core.internal.jobs.Worker.run(Worker.java:58)
      00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
      00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
      00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................
      000000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
      000000b0: 0000 0

      1. DerbyDiag.java
        2 kB
        Kathey Marsden
      2. derby.log
        87 kB
        Shahbaz
      3. DerbyIssues_DBSource.zip
        3.03 MB
        Shahbaz
      4. modules.properties
        5 kB
        Shahbaz
      5. derby.log
        42 kB
        Shahbaz
      6. SingleThread Test.zip
        190 kB
        Shahbaz
      7. DB_10.4logs.zip
        9 kB
        Shahbaz
      8. derby.log
        254 kB
        Shahbaz
      9. hibernate.cfg.xml
        5 kB
        Shahbaz
      10. hibernate.cfg.xml
        2 kB
        Shahbaz
      11. hibernate.cfg.xml
        2 kB
        Shahbaz
      12. derby.log
        1.51 MB
        Shahbaz

        Issue Links

          Activity

          Hide
          Rick Hillegas added a comment -

          This is a serious problem but our experts have not been able to reproduce it. There has been no substantive activity on this issue for more than 2 years. I am closing it as Not Reproducible. The issue can be re-opened if people want to continue work on it.

          Show
          Rick Hillegas added a comment - This is a serious problem but our experts have not been able to reproduce it. There has been no substantive activity on this issue for more than 2 years. I am closing it as Not Reproducible. The issue can be re-opened if people want to continue work on it.
          Hide
          Rick Hillegas added a comment -

          Re-categorizing this as a data corruption, which is more serious than a crash.

          Show
          Rick Hillegas added a comment - Re-categorizing this as a data corruption, which is more serious than a crash.
          Hide
          Kathey Marsden added a comment -

          Changing urgency to normal. Unfortunately we don't have a reproduction and can't hold up 10.5.2 until we get one. As soon as we get a reproduction I think the Urgency should be restored for the next release.

          Show
          Kathey Marsden added a comment - Changing urgency to normal. Unfortunately we don't have a reproduction and can't hold up 10.5.2 until we get one. As soon as we get a reproduction I think the Urgency should be restored for the next release.
          Hide
          Tiago R. Espinha added a comment -

          Triaged for 10.5.2. Checked Crash and Seen in production. Changed components to Miscellaneous.

          Show
          Tiago R. Espinha added a comment - Triaged for 10.5.2. Checked Crash and Seen in production. Changed components to Miscellaneous.
          Hide
          Kathey Marsden added a comment -

          Are you still able to reproduce this corruption? If so you can try to narrow down where it is happening and determine if it is multiple ClassLoaders by calling the attached DerbyDiag.checkTables() at various points in your program. Please pass in the same connection you used for database operations before you closed it. The method

          • Prints the ClassLoader. Check and see if this ClassLoader is different at different points of execution.
          • Checks the tables. (If you run with a SANE build you will get a better check.
          • Prints all thread stack traces. Check to see if you have multiple rawStore daemon threads. You should have only one per database.
          Show
          Kathey Marsden added a comment - Are you still able to reproduce this corruption? If so you can try to narrow down where it is happening and determine if it is multiple ClassLoaders by calling the attached DerbyDiag.checkTables() at various points in your program. Please pass in the same connection you used for database operations before you closed it. The method Prints the ClassLoader. Check and see if this ClassLoader is different at different points of execution. Checks the tables. (If you run with a SANE build you will get a better check. Prints all thread stack traces. Check to see if you have multiple rawStore daemon threads. You should have only one per database.
          Hide
          Shahbaz added a comment -

          It looks like its getting difficult to reproduce it by you guys, can you help us by tracking derby logs to identify the problem area or what all we can do to avoid the same.

          Show
          Shahbaz added a comment - It looks like its getting difficult to reproduce it by you guys, can you help us by tracking derby logs to identify the problem area or what all we can do to avoid the same.
          Hide
          sunil kumar K C added a comment -

          Hi Kristian,
          Thanks for the efforts your taking to solve this issue. I hope you have instaled both the exe's we gave. usually the corrupton happens when we restart the second exe( ACM) and restart the server number of times. The corruption is very random as it can occur in first restart or may be after 5 th or 10th. We have tried in different machines and it happens everyplace even though the frequency is different. i hope you can see 3 Derby db s in the installation folder ( Arena/db).

          thanks
          Sunil

          Show
          sunil kumar K C added a comment - Hi Kristian, Thanks for the efforts your taking to solve this issue. I hope you have instaled both the exe's we gave. usually the corrupton happens when we restart the second exe( ACM) and restart the server number of times. The corruption is very random as it can occur in first restart or may be after 5 th or 10th. We have tried in different machines and it happens everyplace even though the frequency is different. i hope you can see 3 Derby db s in the installation folder ( Arena/db). thanks Sunil
          Hide
          Kristian Waagan added a comment -

          I tried to reproduce, but I'm unable to do so.
          Given you have tried with the client driver and didn't see any problems, one should think the hardware is okay.
          In any case, have you tried the repro on at least two different computers?

          If so, maybe the problem is timing related and is not as easily hit on my machine.
          Any more hints on how I can trigger the corruption?

          So far, I've restarted (single action) and stopped/started the Arena service lots of times and also logged in with the monitoring tool.

          Show
          Kristian Waagan added a comment - I tried to reproduce, but I'm unable to do so. Given you have tried with the client driver and didn't see any problems, one should think the hardware is okay. In any case, have you tried the repro on at least two different computers? If so, maybe the problem is timing related and is not as easily hit on my machine. Any more hints on how I can trigger the corruption? So far, I've restarted (single action) and stopped/started the Arena service lots of times and also logged in with the monitoring tool.
          Hide
          Shahbaz added a comment -

          Thanks Kristian for your concern and attention,

          1)
          We are not using derby plugin "org.apache.derby.core_10.1.2.1", and using only jar, which you found in "./plugins/ae.sphere.arena.database.common_10.4.1/lib/derby.jar"

          2)
          Is your DB corrupted, because for us "Check_table" detects the corruption.

          We will check the FTP server software, now lots of guys complaining about the same.

          Show
          Shahbaz added a comment - Thanks Kristian for your concern and attention, 1) We are not using derby plugin "org.apache.derby.core_10.1.2.1", and using only jar, which you found in "./plugins/ae.sphere.arena.database.common_10.4.1/lib/derby.jar" 2) Is your DB corrupted, because for us "Check_table" detects the corruption. We will check the FTP server software, now lots of guys complaining about the same.
          Hide
          Kristian Waagan added a comment -

          1) I find two. I'm asking because I'm wondering if a class loader issue could be the cause. That is very hard for me to understand, since I don't know your software.
          $ find . -name 'derby.jar'
          ./plugins/ae.sphere.arena.database.common_10.4.1/lib/derby.jar
          ./plugins/org.apache.derby.core_10.1.2.1/derby.jar

          2) Okay, so the corruption happens to different tables/indexes.
          I also notice that running SYSCS_UTIL.SYSCS_CHECK_TABLE doesn't detect the corruption (at least in my case).

          Regarding the FTP issue, BulletProofFTP (a standalone FTP client) couldn't see the files. Maybe the FTP server software is a bit peculiar...

          Show
          Kristian Waagan added a comment - 1) I find two. I'm asking because I'm wondering if a class loader issue could be the cause. That is very hard for me to understand, since I don't know your software. $ find . -name 'derby.jar' ./plugins/ae.sphere.arena.database.common_10.4.1/lib/derby.jar ./plugins/org.apache.derby.core_10.1.2.1/derby.jar 2) Okay, so the corruption happens to different tables/indexes. I also notice that running SYSCS_UTIL.SYSCS_CHECK_TABLE doesn't detect the corruption (at least in my case). Regarding the FTP issue, BulletProofFTP (a standalone FTP client) couldn't see the files. Maybe the FTP server software is a bit peculiar...
          Hide
          Shahbaz added a comment -

          1) There is only one copy of the Derby, there are two "exe" files which needs to be installed. Derby jar you can find in the <installation directory>/plugins/ae.sphere.arena.database.common

          2) Its not always the same table, its very random.

          You may not get the files using browser but you can use either command prompt or any FTP client software to get this.

          Also its very difficult to repro it using standard java program.

          Show
          Shahbaz added a comment - 1) There is only one copy of the Derby, there are two "exe" files which needs to be installed. Derby jar you can find in the <installation directory>/plugins/ae.sphere.arena.database.common 2) Its not always the same table, its very random. You may not get the files using browser but you can use either command prompt or any FTP client software to get this. Also its very difficult to repro it using standard java program.
          Hide
          Kristian Waagan added a comment -

          Two questions;
          1) There are two versions of Derby in the bundle (according to the directory names). Is this intentional?
          2) Is the corrupted table always CONF_DEVICE? Or an index for this table?

          BTW: I also had the same issue as Kathey, where my FTP client couldn't see any files. Using a different client worked around the problem.

          Show
          Kristian Waagan added a comment - Two questions; 1) There are two versions of Derby in the bundle (according to the directory names). Is this intentional? 2) Is the corrupted table always CONF_DEVICE? Or an index for this table? BTW: I also had the same issue as Kathey, where my FTP client couldn't see any files. Using a different client worked around the problem.
          Hide
          Kathey Marsden added a comment -

          One thing to keep in mind is that this is an all volunteer effort, so there is no guarantee that someone will look at your issue. That said, this seems a serious issue that I and I am sure others would like to see fixed. One thing that caused me to back off of looking at the repro was that I had to install software on my pc and I wasn't sure what impact that might have. If I had a junk pc to install it on I would, but I don't right now. Your next step might be to try to create a java program based on what your application is doing and see if you can reproduce the problem. That way you can post the java source so that developers can compile it themselves. Then you'll have a greater chance of someone taking a look and being able to resolve the issue.

          Show
          Kathey Marsden added a comment - One thing to keep in mind is that this is an all volunteer effort, so there is no guarantee that someone will look at your issue. That said, this seems a serious issue that I and I am sure others would like to see fixed. One thing that caused me to back off of looking at the repro was that I had to install software on my pc and I wasn't sure what impact that might have. If I had a junk pc to install it on I would, but I don't right now. Your next step might be to try to create a java program based on what your application is doing and see if you can reproduce the problem. That way you can post the java source so that developers can compile it themselves. Then you'll have a greater chance of someone taking a look and being able to resolve the issue.
          Hide
          sunil kumar K C added a comment -

          Hi Derby team,
          As you people suggested we have uploaded the reproducible test case. But till now there is no response from you. For us its "BLOCKER" and we cant move fwd with out solving this. Please do something for this..........................

          Show
          sunil kumar K C added a comment - Hi Derby team, As you people suggested we have uploaded the reproducible test case. But till now there is no response from you. For us its "BLOCKER" and we cant move fwd with out solving this. Please do something for this..........................
          Hide
          Shahbaz added a comment -

          Guys, any luck on the same as we are able to re-generate this problem quickly.

          Show
          Shahbaz added a comment - Guys, any luck on the same as we are able to re-generate this problem quickly.
          Hide
          Kathey Marsden added a comment -

          Thanks. I can see them from the ftp prompt. For some reason I couldn't access them from my browser. I think, however, because of the size, potential impact to my system and license issues, I won't be installing them.

          Show
          Kathey Marsden added a comment - Thanks. I can see them from the ftp prompt. For some reason I couldn't access them from my browser. I think, however, because of the size, potential impact to my system and license issues, I won't be installing them.
          Hide
          sunil kumar K C added a comment -

          Hi Kathey,
          The files are already there . i am able to see the files through the ftp from command prompt

          ftp> open ftp.sphere.ae
          Connected to sphere.ae.
          220 Microsoft FTP Service
          User (sphere.ae:(none)): anmderby
          331 Password required for anmderby.
          Password:
          230 User anmderby logged in.
          ftp> dir
          200 PORT command successful.
          150 Opening ASCII mode data connection for /bin/ls.
          10-13-08 09:27PM 30148639 ACM_1.1.0.exe
          10-13-08 10:39PM 197208650 ANM_2.3.8.exe
          10-13-08 09:27PM <DIR> db
          10-13-08 09:05PM 1459 installation and testing procedure.txt
          10-13-08 09:05PM 4318 wrapper.conf
          226 Transfer complete.
          ftp: 283 bytes received in 0.00Seconds 70.75Kbytes/sec.
          ftp> ls
          200 PORT command successful.
          150 Opening ASCII mode data connection for file list.
          ACM_1.1.0.exe
          ANM_2.3.8.exe
          db
          installation and testing procedure.txt
          wrapper.conf
          226 Transfer complete.
          ftp: 88 bytes received in 0.00Seconds 29.33Kbytes/sec.
          ftp>

          Show
          sunil kumar K C added a comment - Hi Kathey, The files are already there . i am able to see the files through the ftp from command prompt ftp> open ftp.sphere.ae Connected to sphere.ae. 220 Microsoft FTP Service User (sphere.ae:(none)): anmderby 331 Password required for anmderby. Password: 230 User anmderby logged in. ftp> dir 200 PORT command successful. 150 Opening ASCII mode data connection for /bin/ls. 10-13-08 09:27PM 30148639 ACM_1.1.0.exe 10-13-08 10:39PM 197208650 ANM_2.3.8.exe 10-13-08 09:27PM <DIR> db 10-13-08 09:05PM 1459 installation and testing procedure.txt 10-13-08 09:05PM 4318 wrapper.conf 226 Transfer complete. ftp: 283 bytes received in 0.00Seconds 70.75Kbytes/sec. ftp> ls 200 PORT command successful. 150 Opening ASCII mode data connection for file list. ACM_1.1.0.exe ANM_2.3.8.exe db installation and testing procedure.txt wrapper.conf 226 Transfer complete. ftp: 88 bytes received in 0.00Seconds 29.33Kbytes/sec. ftp>
          Hide
          Kathey Marsden added a comment -

          I'm not seeing any files at that ftp address.

          Show
          Kathey Marsden added a comment - I'm not seeing any files at that ftp address.
          Hide
          sunil kumar K C added a comment -

          We are still facing the db corruption issue and it blocks most of our product installations. I have created an FTP site and uploaded the repro with instructions to do the test.kindly make use of it and give us some solution for this issue. I guess somehow the synchronization between the page file and memory container info goes wrong which results invalid page entries on the disk. As a result when we restart the application the corrupted page loads into the memory resulting in entire db crash.I can give you more information regarding our application set up if you need.
          FTP details
          address- ftp.sphere.ae
          username - anmderby
          password - 12345

          Thanks and regards
          Sunil

          Show
          sunil kumar K C added a comment - We are still facing the db corruption issue and it blocks most of our product installations. I have created an FTP site and uploaded the repro with instructions to do the test.kindly make use of it and give us some solution for this issue. I guess somehow the synchronization between the page file and memory container info goes wrong which results invalid page entries on the disk. As a result when we restart the application the corrupted page loads into the memory resulting in entire db crash.I can give you more information regarding our application set up if you need. FTP details address- ftp.sphere.ae username - anmderby password - 12345 Thanks and regards Sunil
          Hide
          Shahbaz added a comment -

          As suggected by Knut, I checked with latest build, still the same problem in the same place.

          Show
          Shahbaz added a comment - As suggected by Knut, I checked with latest build, still the same problem in the same place.
          Hide
          Shahbaz added a comment -

          Just wondering, we have 2 databases arena and networkManager, but in logs we are getting only db/arena logs, which can be seen in previous attached derby log file, athough both dbs are initialized and working.

          Show
          Shahbaz added a comment - Just wondering, we have 2 databases arena and networkManager, but in logs we are getting only db/arena logs, which can be seen in previous attached derby log file, athough both dbs are initialized and working.
          Hide
          Shahbaz added a comment -

          Hi,

          Sorry guys couldn't come on this issue, as I was busy. I have started logging of Derby, and also found the issue. I have attached the file for your reference.

          Show
          Shahbaz added a comment - Hi, Sorry guys couldn't come on this issue, as I was busy. I have started logging of Derby, and also found the issue. I have attached the file for your reference.
          Hide
          Knut Anders Hatlen added a comment -

          It could be worth a try to check if the fix for DERBY-3726 made any difference. The fix is available if you check out and build the latest development sources (trunk), the 10.4 branch or 10.3 branch. Alternatively, if you don't want to build it yourself, the jar files used in the nightly regression tests can be used to test it. See http://dbtg.thresher.com/derby/bits/.

          Show
          Knut Anders Hatlen added a comment - It could be worth a try to check if the fix for DERBY-3726 made any difference. The fix is available if you check out and build the latest development sources (trunk), the 10.4 branch or 10.3 branch. Alternatively, if you don't want to build it yourself, the jar files used in the nightly regression tests can be used to test it. See http://dbtg.thresher.com/derby/bits/ .
          Hide
          Mike Matrigali added a comment -

          Without more info or a reproducible case I have run out of ideas on this one.

          I looked through the source you posted and nothing jumped out as wrong. Running shutdown as part of a jvmhook
          does put the code in probably an untested area of the JVM. As I read it all shutdown hook things runn concurrently
          so maybe there is a timing thing there. I started wondering if any of the cross class loader locking that we count
          on in XP stop working as various shutdown stuff happens. For instance at what point during jvm shutdown does
          the open file we have on the db lock file actually get closed? I could not tell from the code if there are actually multiple class loaders involved in your app accessing a single derby db.

          If I had a reproducible case I would run the application with the following properties:
          derby.stream.error.logSeverityLevel=0
          derby.storage.keepTransactionLog=true
          derby.language.logStatementText=true
          derby.infolog.append=true

          This will require a lot of disk space. The keepTransactionLog property will keep all the transaction log files so after the fact one can see the exact history of the updating transactions during the test run. logStatementText helps with tracking the non-updating queries and will generate a huge derby.log.

          Show
          Mike Matrigali added a comment - Without more info or a reproducible case I have run out of ideas on this one. I looked through the source you posted and nothing jumped out as wrong. Running shutdown as part of a jvmhook does put the code in probably an untested area of the JVM. As I read it all shutdown hook things runn concurrently so maybe there is a timing thing there. I started wondering if any of the cross class loader locking that we count on in XP stop working as various shutdown stuff happens. For instance at what point during jvm shutdown does the open file we have on the db lock file actually get closed? I could not tell from the code if there are actually multiple class loaders involved in your app accessing a single derby db. If I had a reproducible case I would run the application with the following properties: derby.stream.error.logSeverityLevel=0 derby.storage.keepTransactionLog=true derby.language.logStatementText=true derby.infolog.append=true This will require a lot of disk space. The keepTransactionLog property will keep all the transaction log files so after the fact one can see the exact history of the updating transactions during the test run. logStatementText helps with tracking the non-updating queries and will generate a huge derby.log.
          Hide
          Shahbaz added a comment -

          Please find the attached classes for all DB start ups and shut down. We have two DBs Arena and NetworkManager.

          ae.sphere.arena.networkManager.db.NetworkManagerDBStartup is the startup class of NM db

          ae.sphere.arena.arena.db.ArenaDBStartup is the startup class of Arena db

          ae.sphere.arena.database.management.shutdown.AddDBShutdownThreadHook is where we are adding jvm hooks for shutdown's.

          From hibernate level we are clearing sessions individualy but fron derby level its single which can be found in shutdownDB() menthod of ae.sphere.arena.database.common.management.DBManager class.

          Show
          Shahbaz added a comment - Please find the attached classes for all DB start ups and shut down. We have two DBs Arena and NetworkManager. ae.sphere.arena.networkManager.db.NetworkManagerDBStartup is the startup class of NM db ae.sphere.arena.arena.db.ArenaDBStartup is the startup class of Arena db ae.sphere.arena.database.management.shutdown.AddDBShutdownThreadHook is where we are adding jvm hooks for shutdown's. From hibernate level we are clearing sessions individualy but fron derby level its single which can be found in shutdownDB() menthod of ae.sphere.arena.database.common.management.DBManager class.
          Hide
          Mike Matrigali added a comment - - edited

          could you remove the db shutdown hook and see if you can reproduce the bug.

          Is it possible that the shutdown hook is running in a different class loader from the class loader that started the db/connects to the db? I didn't think DERBY-700 applied on windows, but doing the above experiment will
          give some important information.

          It may help to post the code for all of your shutdown hooks. From before it looks like there is 1 for shutting down
          the derby SYSTEM, and thus all the dbs. You describe another that shutdowns the clents - is there one or maybe one per db? Any others that are related to derby?

          Show
          Mike Matrigali added a comment - - edited could you remove the db shutdown hook and see if you can reproduce the bug. Is it possible that the shutdown hook is running in a different class loader from the class loader that started the db/connects to the db? I didn't think DERBY-700 applied on windows, but doing the above experiment will give some important information. It may help to post the code for all of your shutdown hooks. From before it looks like there is 1 for shutting down the derby SYSTEM, and thus all the dbs. You describe another that shutdowns the clents - is there one or maybe one per db? Any others that are related to derby?
          Hide
          Mike Matrigali added a comment -

          > 1. Corrupted DB can not be used in any case, so we are using fresh db but once
          generated.
          great, that is fine.
          > 2. Online back up component is on but it never happened, DB gets corrupted bef
          ore back up. so it never happened during tests.
          ok, now I won't worry about looking at the online backup code.
          > 3. Shutdown process, I already explained in previous posts and startup is simp
          le, we just start the dbs, start the connections and start the normal work for a
          pplication. We do the same work to reproduce the same, the only difference is th
          at some times it appears quickly and some times it needs some 6-7 restarts. Our
          application doesn't know when its corrupted, but as user we can know with functi
          onality failures and logs. Corrupted tables are not consistent meaning some time
          s some tables and some times another tables.

          > 4. Do you want to say that corruption happens during running of application?

          It is all a guess at this point. But often with these types a problems the
          corruption happens at an earlier time either as part of the application or
          part of a previous shutdown. This problem has the feel of the page being
          fine in the buffer cache, but somehow being corrupted as it is being written
          out. Because of the cache the application may run just fine accessing the
          "corrupted on disk page" because the good copy is still in cache.

          > 5. I can send you application exe with DB, which is around 200 MB, as sample i
          s not able to corrupt the same. for the same we can create one ftp account for y
          ou guys to download and test from their.

          if you can set up a download site and instructions on how to reproduce, seems
          much more likely someone will be able to figure out what is going on.
          If the 200mb example does not reproduce the issue then that does not seem
          helpful. Go ahead and set up ftp of case that reproduces the problem,
          along with details on how to reproduce.

          Show
          Mike Matrigali added a comment - > 1. Corrupted DB can not be used in any case, so we are using fresh db but once generated. great, that is fine. > 2. Online back up component is on but it never happened, DB gets corrupted bef ore back up. so it never happened during tests. ok, now I won't worry about looking at the online backup code. > 3. Shutdown process, I already explained in previous posts and startup is simp le, we just start the dbs, start the connections and start the normal work for a pplication. We do the same work to reproduce the same, the only difference is th at some times it appears quickly and some times it needs some 6-7 restarts. Our application doesn't know when its corrupted, but as user we can know with functi onality failures and logs. Corrupted tables are not consistent meaning some time s some tables and some times another tables. > 4. Do you want to say that corruption happens during running of application? It is all a guess at this point. But often with these types a problems the corruption happens at an earlier time either as part of the application or part of a previous shutdown. This problem has the feel of the page being fine in the buffer cache, but somehow being corrupted as it is being written out. Because of the cache the application may run just fine accessing the "corrupted on disk page" because the good copy is still in cache. > 5. I can send you application exe with DB, which is around 200 MB, as sample i s not able to corrupt the same. for the same we can create one ftp account for y ou guys to download and test from their. if you can set up a download site and instructions on how to reproduce, seems much more likely someone will be able to figure out what is going on. If the 200mb example does not reproduce the issue then that does not seem helpful. Go ahead and set up ftp of case that reproduces the problem, along with details on how to reproduce.
          Hide
          Shahbaz added a comment -

          1. Corrupted DB can not be used in any case, so we are using fresh db but once generated.
          2. Online back up component is on but it never happened, DB gets corrupted before back up. so it never happened during tests.
          3. Shutdown process, I already explained in previous posts and startup is simple, we just start the dbs, start the connections and start the normal work for application. We do the same work to reproduce the same, the only difference is that some times it appears quickly and some times it needs some 6-7 restarts. Our application doesn't know when its corrupted, but as user we can know with functionality failures and logs. Corrupted tables are not consistent meaning some times some tables and some times another tables.
          4. Do you want to say that corruption happens during running of application?
          5. I can send you application exe with DB, which is around 200 MB, as sample is not able to corrupt the same. for the same we can create one ftp account for you guys to download and test from their.

          Show
          Shahbaz added a comment - 1. Corrupted DB can not be used in any case, so we are using fresh db but once generated. 2. Online back up component is on but it never happened, DB gets corrupted before back up. so it never happened during tests. 3. Shutdown process, I already explained in previous posts and startup is simple, we just start the dbs, start the connections and start the normal work for application. We do the same work to reproduce the same, the only difference is that some times it appears quickly and some times it needs some 6-7 restarts. Our application doesn't know when its corrupted, but as user we can know with functionality failures and logs. Corrupted tables are not consistent meaning some times some tables and some times another tables. 4. Do you want to say that corruption happens during running of application? 5. I can send you application exe with DB, which is around 200 MB, as sample is not able to corrupt the same. for the same we can create one ftp account for you guys to download and test from their.
          Hide
          Mike Matrigali added a comment -

          If you create the db and then use a copy of it for each iteration then that is ok. The nature of the corruption is that once it happens it will not get fixed, so if you use a broken one for the next test, the test does not tell us anything. My assumption
          of what is happening is that the page actually gets corrupted before it gets recognized. This may be happening one or
          more complete startup/shutdown phases before it actually gets noticed.

          I didn't completely understand your answer about online backup. Was an online backup ever taken during the
          reproduction, anytime from starting with the fresh db until the corruption happens.

          It is not clear to me your application knows exactly when the corruption happens. You still have not described what work is done in each startup/shutdown phase. If it is exactly the same work then it is likely it does know. But if there is some
          random/test work it may be that you don't see the page in one iteration. You could run a consistency check in between
          each phase to definitely tell when the corruption happens.

          The best case would be to attach the application/db to this jira for anyone to reproduce. What is the zipped size of the jars and db?

          Show
          Mike Matrigali added a comment - If you create the db and then use a copy of it for each iteration then that is ok. The nature of the corruption is that once it happens it will not get fixed, so if you use a broken one for the next test, the test does not tell us anything. My assumption of what is happening is that the page actually gets corrupted before it gets recognized. This may be happening one or more complete startup/shutdown phases before it actually gets noticed. I didn't completely understand your answer about online backup. Was an online backup ever taken during the reproduction, anytime from starting with the fresh db until the corruption happens. It is not clear to me your application knows exactly when the corruption happens. You still have not described what work is done in each startup/shutdown phase. If it is exactly the same work then it is likely it does know. But if there is some random/test work it may be that you don't see the page in one iteration. You could run a consistency check in between each phase to definitely tell when the corruption happens. The best case would be to attach the application/db to this jira for anyone to reproduce. What is the zipped size of the jars and db?
          Hide
          Shahbaz added a comment -

          Online backup is not working at the time of corruption. We are using Java 1.5 from last some tests.

          Can we send you whole application installers with DB to reproduce the same at your end. As it will be quicker?

          Attach are modules.property file.

          Everytime we are not generating new db, for example for this changes in properties tests, we used our 10.4 generated db and performed test. Is it cruicial to generate the same everytime, as I dont think these changes should have any affect on the same.

          Show
          Shahbaz added a comment - Online backup is not working at the time of corruption. We are using Java 1.5 from last some tests. Can we send you whole application installers with DB to reproduce the same at your end. As it will be quicker? Attach are modules.property file. Everytime we are not generating new db, for example for this changes in properties tests, we used our 10.4 generated db and performed test. Is it cruicial to generate the same everytime, as I dont think these changes should have any affect on the same.
          Hide
          Mike Matrigali added a comment -

          After the first startup/shutdown phase, does your reproduction test create tables in the subsequent startup/shutdowns?

          Show
          Mike Matrigali added a comment - After the first startup/shutdown phase, does your reproduction test create tables in the subsequent startup/shutdowns?
          Hide
          Mike Matrigali added a comment -

          Your latest run came up with a slightly different kind of error, a checksum error rather than the consistent array out
          of bounds from previous runs. I don't know if that is signicant or not. If you have time could you run the repro of few more
          times with nio disabled and post the logs - maybe give them different names to make it clear which is which.
          Just to see if checksum is always the error in this case. The difference is that the header info is readable in this latest
          case and then we check the checksum, but in previous case we can't even read the header. Given the latest problem
          maybe there is bad synchronization in some path of the base i/o code, and not the nio code.

          did you ever get a chance to turn off all backups during your repro case?

          Could you post the modules.properties you ran your last test with. If this did turn off nio, and it was the only one in
          your path then the issue is in some other part of code than I have be guessing at. Again the assumption is that
          each time you post results your test run starts with a new created db.

          I am not aware of any issues with jdk15 or jdk16 that would cause this issue. Just to be specific can you post exact
          jdk versions of the jvm's you are running the tests with when you post your results. For instance is it the sun jdk16?
          java -version will give you the output i am interested in. I don't really think the jvm is the issue. you could try latest
          test case, in each instance creating new db's for each test run agains jdk14, jdk15, and jdk16 to make sure.

          Show
          Mike Matrigali added a comment - Your latest run came up with a slightly different kind of error, a checksum error rather than the consistent array out of bounds from previous runs. I don't know if that is signicant or not. If you have time could you run the repro of few more times with nio disabled and post the logs - maybe give them different names to make it clear which is which. Just to see if checksum is always the error in this case. The difference is that the header info is readable in this latest case and then we check the checksum, but in previous case we can't even read the header. Given the latest problem maybe there is bad synchronization in some path of the base i/o code, and not the nio code. did you ever get a chance to turn off all backups during your repro case? Could you post the modules.properties you ran your last test with. If this did turn off nio, and it was the only one in your path then the issue is in some other part of code than I have be guessing at. Again the assumption is that each time you post results your test run starts with a new created db. I am not aware of any issues with jdk15 or jdk16 that would cause this issue. Just to be specific can you post exact jdk versions of the jvm's you are running the tests with when you post your results. For instance is it the sun jdk16? java -version will give you the output i am interested in. I don't really think the jvm is the issue. you could try latest test case, in each instance creating new db's for each test run agains jdk14, jdk15, and jdk16 to make sure.
          Hide
          Shahbaz added a comment -

          After commenting the same as suggested, we found the exceptions as in attached logs.

          Show
          Shahbaz added a comment - After commenting the same as suggested, we found the exceptions as in attached logs.
          Hide
          Knut Anders Hatlen added a comment -

          That line is only part of modules.properties in the source tree. If you modify modules.properties in derby.jar, just deleting the other properties Mike mentioned should be OK.

          If this is the same issue as DERBY-3606 (same reporter, same symptoms), I'd assume the NIO changes are not causing the problem, as that bug was reported against 10.2.2.0. The NIO code was introduced in 10.3.1.4.

          Show
          Knut Anders Hatlen added a comment - That line is only part of modules.properties in the source tree. If you modify modules.properties in derby.jar, just deleting the other properties Mike mentioned should be OK. If this is the same issue as DERBY-3606 (same reporter, same symptoms), I'd assume the NIO changes are not causing the problem, as that bug was reported against 10.2.2.0. The NIO code was introduced in 10.3.1.4.
          Hide
          Shahbaz added a comment -

          We will do the same test and let you know, but we couldnt find "cloudscape.config.rawStore.data.genericJ4" property in module.properties in derby.jar

          Please advise.

          Show
          Shahbaz added a comment - We will do the same test and let you know, but we couldnt find "cloudscape.config.rawStore.data.genericJ4" property in module.properties in derby.jar Please advise.
          Hide
          Shahbaz added a comment -

          For your information, we are using Java 1.5 and 1.6. Is there any expected problem using the same?

          Show
          Shahbaz added a comment - For your information, we are using Java 1.5 and 1.6. Is there any expected problem using the same?
          Hide
          Mike Matrigali added a comment -

          if you can't come up with a repro, are you able to build/debug/run derby itself. The following suggestion would help identify if nio
          is still the problem:

          One thing I know was changed in that area of the code in that period, is that RAFContainer started using java.nio.* to allow parallel reads and writes on JVMs that support nio. It is possible to force the use of the old mechanism by deleting or commenting out the following lines in java/engine/org/apache/derby/modules.properties and rebuilding derby.jar:

          1. store data using a StorageFactory
          2. Enhanced version using NIO API; requires Java 1.4
            derby.module.rawStore.data.genericJ4=org.apache.derby.impl.store.raw.data.BaseDataFileFactoryJ4
            derby.env.jdk.rawStore.data.genericJ4=4
            derby.env.classes.rawStore.data.genericJ4=java.nio.Buffer
            cloudscape.config.rawStore.data.genericJ4=derby

          It would be great if you could make that change and rerun your test.

          Show
          Mike Matrigali added a comment - if you can't come up with a repro, are you able to build/debug/run derby itself. The following suggestion would help identify if nio is still the problem: One thing I know was changed in that area of the code in that period, is that RAFContainer started using java.nio.* to allow parallel reads and writes on JVMs that support nio. It is possible to force the use of the old mechanism by deleting or commenting out the following lines in java/engine/org/apache/derby/modules.properties and rebuilding derby.jar: store data using a StorageFactory Enhanced version using NIO API; requires Java 1.4 derby.module.rawStore.data.genericJ4=org.apache.derby.impl.store.raw.data.BaseDataFileFactoryJ4 derby.env.jdk.rawStore.data.genericJ4=4 derby.env.classes.rawStore.data.genericJ4=java.nio.Buffer cloudscape.config.rawStore.data.genericJ4=derby It would be great if you could make that change and rerun your test.
          Hide
          Shahbaz added a comment -

          Its Intel Core 2 CPU 6300, 1.8 GHz, with Win XP SP2. We are mostly updating the rows very frequently. Deleting is not that frequent but update/Insert/select is very frequent. and at the time of service restarts also the transactions must be on.

          Show
          Shahbaz added a comment - Its Intel Core 2 CPU 6300, 1.8 GHz, with Win XP SP2. We are mostly updating the rows very frequently. Deleting is not that frequent but update/Insert/select is very frequent. and at the time of service restarts also the transactions must be on.
          Hide
          Mike Matrigali added a comment -

          what hardware are you running the test on? Mostly I am looking for number of cpu's and number of cores on the cpu's to again understand the possible concurrency going on. Thanks for running the single thread test, I will look at it. Even if from your app
          point of view it is single threaded, derby may use multiple threads per db to take care of background things like checkpoint and
          delete row processing. That is why I wanted to know a little more about the actions your app takes when reproducing the problem.

          Show
          Mike Matrigali added a comment - what hardware are you running the test on? Mostly I am looking for number of cpu's and number of cores on the cpu's to again understand the possible concurrency going on. Thanks for running the single thread test, I will look at it. Even if from your app point of view it is single threaded, derby may use multiple threads per db to take care of background things like checkpoint and delete row processing. That is why I wanted to know a little more about the actions your app takes when reproducing the problem.
          Hide
          Shahbaz added a comment -

          1. Default online backup is daily, but I dont think that it is running at the same time of corruption, as timings are completely different, but we will do the test and let you know.

          2. You are right in getting the shutdown and start up processes. What do you mean by "clients"?. Using hibernate, we are closing all the connections and all, befor derby shutdown.

          3. Shutdown/Startup process is consistant.

          4. Derby properties

          System.setProperty("derby.stream.error.file", "logs/derby.log");
          System.setProperty("derby.stream.error.logSeverityLevel", "0");
          System.setProperty("derby.locks.monitor","true");
          System.setProperty("derby.locks.deadlockTrace","true");
          System.setProperty("derby.locks.deadlockTimeout","20");

          Show
          Shahbaz added a comment - 1. Default online backup is daily, but I dont think that it is running at the same time of corruption, as timings are completely different, but we will do the test and let you know. 2. You are right in getting the shutdown and start up processes. What do you mean by "clients"?. Using hibernate, we are closing all the connections and all, befor derby shutdown. 3. Shutdown/Startup process is consistant. 4. Derby properties System.setProperty("derby.stream.error.file", "logs/derby.log"); System.setProperty("derby.stream.error.logSeverityLevel", "0"); System.setProperty("derby.locks.monitor","true"); System.setProperty("derby.locks.deadlockTrace","true"); System.setProperty("derby.locks.deadlockTimeout","20");
          Hide
          Shahbaz added a comment -

          Oops, we tried to create single thread test, making sure that at any point only one thread is active, still we found same issue, using 10.4.

          Show
          Shahbaz added a comment - Oops, we tried to create single thread test, making sure that at any point only one thread is active, still we found same issue, using 10.4.
          Hide
          Mike Matrigali added a comment -

          Thanks for the info, without a repro I have been inspecting the code and anything that you can tell me about the app
          helps direct that effort. Of course if you can come up with a repro that could be run it is much more likely someone will
          be able to find the issue. If you can't give a repro I am going to continue to ask questions about the app in your environment,
          and maybe something will come of that.

          How big is a zipped copy of the db? Getting a copy of it may help as it would be interesting to look at the state of page 0 in
          the 2 corrupted tables. jira will allow up to 10mb of a zipped attachment. there may be other places at a apache we could
          load a bigger file, just not sure right now. depending on when you enabled online backup the log may include a complete
          update history of the db and looking at it or more likely comparing it with a few other examples of the bug may lead to what
          kinds of things lead to the problem.

          My current assumption is that the problem is caused by some I/O interaction similar to DERBY-3347. Could you describe
          the concurrency in the simplest case that you have been able to reproduce this problem. Basically how many threads
          are involved and do they run concurrently? Things like are the startup/shutdown of the 2 db's done independently on different
          threads? While running how many threads/connections are done doing work in the 2 db's.

          Can you describe when and how often you execute the command that "enables" online backup? It is at this point that derby
          copies a number of database files from the original db to your backup location and it does have some code that insures that
          the page 0 is up to date before the copy. Is it possible to run your test without online backup ever being called just to see it
          the bug still reproduces?

          You mentioned "jvm hook", does this mean you are also shutting down the jvm durring the test run? If so can you describe
          how often, ie. for every shutdown in the derby.log is there also a jvm shutdown? I think this is what you mean by your
          services comment. Is the following what is going on:
          o You start a service and what it does is start up a jvm, it opens the 2 databases, and some set of work is somehow done
          in both the db's (is this work different or the same for each try). Then you stop the service sometime later which shuts down
          the jvm and as part of jvm hooks your specific shutdown stuff happens.

          How are clients shutdown? Is it possible that in progress clients are shutdown by "killing" them somehow? I am not
          familar with hibernate so this could be a part of "the seesion factories (hibernate level) shutdown" ?

          Is any of the work that is done during the tests include deletes, updates to fields included in any index/contraint, or inserts which fail due to duplicate key errors, ddl after the initial create of the database? This is interesting as it may queue background work to reclaim deleted space and thus add another point of concurrency to the work being done.

          Does each startup/shutdown phase always access the same tables, looking to verify that when you get an error on a particular
          table whether it is guaranteed that the table was working in the previous startup/shutdown test phase.

          Another thing I am looking at is a possiblity that exaustion of resources is coming into play, which would explain why it takes multiple db's. Do you set any derby properties as part of your application, if so could you post all the ones that you change. Can you estimate about how many different derby tables might be accessed during one of the startup/shutdown phases of your test? I know this is hard as it should also include how many indexes may be referenced. The 2 resources I am mostly thinking about are the page cache and the open container cache. The page cache defaults to 1000 and the open container cache defaults to 100. Depending on your application (basically concurrent user threads and the background thread used for post commit and checkpoints) we may have multiple open "channels" on each container in the open container cache - i am not exactly sure what
          resource this maps to on windows.

          Show
          Mike Matrigali added a comment - Thanks for the info, without a repro I have been inspecting the code and anything that you can tell me about the app helps direct that effort. Of course if you can come up with a repro that could be run it is much more likely someone will be able to find the issue. If you can't give a repro I am going to continue to ask questions about the app in your environment, and maybe something will come of that. How big is a zipped copy of the db? Getting a copy of it may help as it would be interesting to look at the state of page 0 in the 2 corrupted tables. jira will allow up to 10mb of a zipped attachment. there may be other places at a apache we could load a bigger file, just not sure right now. depending on when you enabled online backup the log may include a complete update history of the db and looking at it or more likely comparing it with a few other examples of the bug may lead to what kinds of things lead to the problem. My current assumption is that the problem is caused by some I/O interaction similar to DERBY-3347 . Could you describe the concurrency in the simplest case that you have been able to reproduce this problem. Basically how many threads are involved and do they run concurrently? Things like are the startup/shutdown of the 2 db's done independently on different threads? While running how many threads/connections are done doing work in the 2 db's. Can you describe when and how often you execute the command that "enables" online backup? It is at this point that derby copies a number of database files from the original db to your backup location and it does have some code that insures that the page 0 is up to date before the copy. Is it possible to run your test without online backup ever being called just to see it the bug still reproduces? You mentioned "jvm hook", does this mean you are also shutting down the jvm durring the test run? If so can you describe how often, ie. for every shutdown in the derby.log is there also a jvm shutdown? I think this is what you mean by your services comment. Is the following what is going on: o You start a service and what it does is start up a jvm, it opens the 2 databases, and some set of work is somehow done in both the db's (is this work different or the same for each try). Then you stop the service sometime later which shuts down the jvm and as part of jvm hooks your specific shutdown stuff happens. How are clients shutdown? Is it possible that in progress clients are shutdown by "killing" them somehow? I am not familar with hibernate so this could be a part of "the seesion factories (hibernate level) shutdown" ? Is any of the work that is done during the tests include deletes, updates to fields included in any index/contraint, or inserts which fail due to duplicate key errors, ddl after the initial create of the database? This is interesting as it may queue background work to reclaim deleted space and thus add another point of concurrency to the work being done. Does each startup/shutdown phase always access the same tables, looking to verify that when you get an error on a particular table whether it is guaranteed that the table was working in the previous startup/shutdown test phase. Another thing I am looking at is a possiblity that exaustion of resources is coming into play, which would explain why it takes multiple db's. Do you set any derby properties as part of your application, if so could you post all the ones that you change. Can you estimate about how many different derby tables might be accessed during one of the startup/shutdown phases of your test? I know this is hard as it should also include how many indexes may be referenced. The 2 resources I am mostly thinking about are the page cache and the open container cache. The page cache defaults to 1000 and the open container cache defaults to 100. Depending on your application (basically concurrent user threads and the background thread used for post commit and checkpoints) we may have multiple open "channels" on each container in the open container cache - i am not exactly sure what resource this maps to on windows.
          Hide
          Shahbaz added a comment -

          1. Yes DB_10.4logs.zip is complete 10.4 (DB and application)
          2. We actualy have 3 dbs, and in this case we were able to repro this problem quickly, so we made the 2 dbs and the above case happened with that only, although not as frequently. As of now we cant have 1 db.
          3. This problem comes only when we do shutdown and startups, so while testing we have to do the same multiple times. We are running our servers as services, so we just start and stop the service to repro the same.
          4. For shut down, we have added this on jvm hook, in that we do following things
          a. get all DBs
          b. close all current sessions, and seesion factories (hibernate level) for each db
          c. finally Derby shutdown using following method, for all dbs

          public boolean shutdownDB(){
          try

          { Bundle b = Platform.getBundle("ae.sphere.arena.database.common"); b.loadClass("org.apache.derby.jdbc.EmbeddedDriver").newInstance(); DriverManager.getConnection("jdbc:derby:;shutdown=true", "xxx", "xxxx"); return false; }

          catch (ClassNotFoundException e)

          { LOG.error(e); } catch (SQLException e) {
          if (e.getSQLState()!= null && (e.getSQLState().equals("08006") || (e.getSQLState().equals("XJ015")))) { LOG.error("it is not error, shutdown successfully ;"+e.getMessage()); return true; }else{ LOG.error(e); }
          } catch (InstantiationException e) { LOG.error(e); }

          catch (IllegalAccessException e)

          { LOG.error(e); }

          return false;
          }

          5. Databases are not encrypted but encrypted data we push.

          6. We have "online" backup and its always on, depends on the schedule set by user.

          7. We tried to prepare the sample for you to reproduce, but while testing we couldnt get the corruption, so no point of senditng it to you, however with full application it is easier to repro the same. Will corrupted DB help you, if yes where and how we can send you, as the db size is big.

          Show
          Shahbaz added a comment - 1. Yes DB_10.4logs.zip is complete 10.4 (DB and application) 2. We actualy have 3 dbs, and in this case we were able to repro this problem quickly, so we made the 2 dbs and the above case happened with that only, although not as frequently. As of now we cant have 1 db. 3. This problem comes only when we do shutdown and startups, so while testing we have to do the same multiple times. We are running our servers as services, so we just start and stop the service to repro the same. 4. For shut down, we have added this on jvm hook, in that we do following things a. get all DBs b. close all current sessions, and seesion factories (hibernate level) for each db c. finally Derby shutdown using following method, for all dbs public boolean shutdownDB(){ try { Bundle b = Platform.getBundle("ae.sphere.arena.database.common"); b.loadClass("org.apache.derby.jdbc.EmbeddedDriver").newInstance(); DriverManager.getConnection("jdbc:derby:;shutdown=true", "xxx", "xxxx"); return false; } catch (ClassNotFoundException e) { LOG.error(e); } catch (SQLException e) { if (e.getSQLState()!= null && (e.getSQLState().equals("08006") || (e.getSQLState().equals("XJ015")))) { LOG.error("it is not error, shutdown successfully ;"+e.getMessage()); return true; }else{ LOG.error(e); } } catch (InstantiationException e) { LOG.error(e); } catch (IllegalAccessException e) { LOG.error(e); } return false; } 5. Databases are not encrypted but encrypted data we push. 6. We have "online" backup and its always on, depends on the schedule set by user. 7. We tried to prepare the sample for you to reproduce, but while testing we couldnt get the corruption, so no point of senditng it to you, however with full application it is easier to repro the same. Will corrupted DB help you, if yes where and how we can send you, as the db size is big.
          Hide
          Mike Matrigali added a comment -

          another code path question, during the repro do you back up the database?

          Show
          Mike Matrigali added a comment - another code path question, during the repro do you back up the database?
          Hide
          Mike Matrigali added a comment -

          just to eliminate another code path - your databases are not encrypted, right?

          Show
          Mike Matrigali added a comment - just to eliminate another code path - your databases are not encrypted, right?
          Hide
          Mike Matrigali added a comment -

          Just to verify, is it true that the 6/11 zip's - DB_10.4logs.zip represent running your application on 10.4 and creating the databases in 10.4. Do the log's represent the entire run of the test? This derby.log did not have the checkpoint errors so if this is the complete run that can probably be eliminated. The subsequent error looks the same as the previous postings - ie. there is some
          corruption in the first page of an index on the Does your app have the ability to run with just one database? And if so can you reproduce this problem with just one database - I am just trying to narrow down the cause.
          The latest is showing corruptions in 1st page of at least 2 indexes (but hard to say exactly what without the db, if there are views involved then could be same table).
          1st:
          index on Arena.USERS
          2nd:
          index on Arena.NOTIFICATION_SCHEDULER

          No other errors in the logs. There are a number of startups and shutdowns. Could you describe how you shutdown the databases?

          Show
          Mike Matrigali added a comment - Just to verify, is it true that the 6/11 zip's - DB_10.4logs.zip represent running your application on 10.4 and creating the databases in 10.4. Do the log's represent the entire run of the test? This derby.log did not have the checkpoint errors so if this is the complete run that can probably be eliminated. The subsequent error looks the same as the previous postings - ie. there is some corruption in the first page of an index on the Does your app have the ability to run with just one database? And if so can you reproduce this problem with just one database - I am just trying to narrow down the cause. The latest is showing corruptions in 1st page of at least 2 indexes (but hard to say exactly what without the db, if there are views involved then could be same table). 1st: index on Arena.USERS 2nd: index on Arena.NOTIFICATION_SCHEDULER No other errors in the logs. There are a number of startups and shutdowns. Could you describe how you shutdown the databases?
          Hide
          Shahbaz added a comment -

          Hi,
          Please find the logs of the same issue using 10.4 generated DB. Two DBs running in embedded mode.

          Show
          Shahbaz added a comment - Hi, Please find the logs of the same issue using 10.4 generated DB. Two DBs running in embedded mode.
          Hide
          Shahbaz added a comment -

          Just want to add that the DB is 10.3 generated, sorry forgot to mention earlier. Now we are trying to migrate the data into 10.4 DB.

          Show
          Shahbaz added a comment - Just want to add that the DB is 10.3 generated, sorry forgot to mention earlier. Now we are trying to migrate the data into 10.4 DB.
          Hide
          Mike Matrigali added a comment -

          my understanding is that the june 8 derby.log represents a complete derby.log starting with newly created databases
          using the 10.4.1.3 derby release. I also assume you can't post the database, or a reproducible case.
          If the above is true then the following reading of the derby.log shows:

          It looks like you boot into 2 different databases. And then there are 4 stacks from failed checkpoints as part of shutdowns
          caused by a call from outside of derby coming in. The checkpoints are failing from:
          java.nio.channels.ClosedChannelException^M
          at sun.nio.ch.FileChannelImpl.ensureOpen(Unknown Source)^M
          at sun.nio.ch.FileChannelImpl.read(Unknown Source)^M

          which I think we have seen from threads getting interrupts sent at them. Unfortunately derby does not log what db the shutdown failures are associated with, nor useful info on what container it is failing on.

          But none of this should cause the subsequent issues, but maybe it is related. Can you run the repro again from scratch (ie. creating new dbs again) and post a new derby.log so we can see if same sequence shows up?

          When the system reboots an ArrayOutOfBounds exception comes while reading a containerheader, leading me to believe that the container header field that holds the borrowed space amount has been corrupted.:

          java.lang.ArrayIndexOutOfBoundsException^M
          at java.lang.System.arraycopy(Native Method)^M
          at org.apache.derby.impl.store.raw.data.AllocPage.ReadContainerInfo(Unknown
          Source)^M
          at org.apache.derby.impl.store.raw.data.FileContainer.readHeader(Unknown Sou
          rce)^M

          Question is if the failed checkpoints somehow led to the corrupted container header.

          Show
          Mike Matrigali added a comment - my understanding is that the june 8 derby.log represents a complete derby.log starting with newly created databases using the 10.4.1.3 derby release. I also assume you can't post the database, or a reproducible case. If the above is true then the following reading of the derby.log shows: It looks like you boot into 2 different databases. And then there are 4 stacks from failed checkpoints as part of shutdowns caused by a call from outside of derby coming in. The checkpoints are failing from: java.nio.channels.ClosedChannelException^M at sun.nio.ch.FileChannelImpl.ensureOpen(Unknown Source)^M at sun.nio.ch.FileChannelImpl.read(Unknown Source)^M which I think we have seen from threads getting interrupts sent at them. Unfortunately derby does not log what db the shutdown failures are associated with, nor useful info on what container it is failing on. But none of this should cause the subsequent issues, but maybe it is related. Can you run the repro again from scratch (ie. creating new dbs again) and post a new derby.log so we can see if same sequence shows up? When the system reboots an ArrayOutOfBounds exception comes while reading a containerheader, leading me to believe that the container header field that holds the borrowed space amount has been corrupted.: java.lang.ArrayIndexOutOfBoundsException^M at java.lang.System.arraycopy(Native Method)^M at org.apache.derby.impl.store.raw.data.AllocPage.ReadContainerInfo(Unknown Source)^M at org.apache.derby.impl.store.raw.data.FileContainer.readHeader(Unknown Sou rce)^M Question is if the failed checkpoints somehow led to the corrupted container header.
          Hide
          Bassel Kh added a comment -

          It is very strange of having these kinds of bugs with such a professional db like Derby.
          Anyways, the error has changed after using Derby 10.4:
          2008-06-11 10:50:37.997 GMT Thread[Worker-60,5,main] (XID = 825230855), (SESSIONID = 29), (DATABASE = db/networkManager), (DRDAID = null), Cleanup action starting
          2008-06-11 10:50:37.997 GMT Thread[Worker-60,5,main] (XID = 825230855), (SESSIONID = 29), (DATABASE = db/networkManager), (DRDAID = null), Failed Statement is: select monitor.status as status , count(monitor.status) as statusCount , sum (monitor.metthresholdtimes) as thresholdTimes from NetworkManager.ACTIVE_MONITOR as monitor where monitor.col1 <> 'Yes' group by monitor.status
          java.lang.ArrayIndexOutOfBoundsException
          at java.lang.System.arraycopy(Native Method)
          at org.apache.derby.impl.store.raw.data.AllocPage.ReadContainerInfo(Unknown Source)
          at org.apache.derby.impl.store.raw.data.FileContainer.readHeader(Unknown Source)
          at org.apache.derby.impl.store.raw.data.RAFContainer.run(Unknown Source)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.apache.derby.impl.store.raw.data.RAFContainer.openContainer(Unknown Source)
          at org.apache.derby.impl.store.raw.data.FileContainer.setIdent(Unknown Source)
          at org.apache.derby.impl.store.raw.data.RAFContainer.setIdentity(Unknown Source)
          at org.apache.derby.impl.services.cache.ConcurrentCache.find(Unknown Source)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(Unknown Source)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(Unknown Source)
          at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Unknown Source)
          at org.apache.derby.impl.store.access.btree.index.B2IFactory.readConglomerate(Unknown Source)
          at org.apache.derby.impl.store.access.RAMAccessManager.conglomCacheFind(Unknown Source)
          at org.apache.derby.impl.store.access.RAMTransaction.findExistingConglomerate(Unknown Source)
          at org.apache.derby.impl.store.access.RAMTransaction.openStoreCost(Unknown Source)
          at org.apache.derby.impl.sql.compile.CompilerContextImpl.getStoreCostController(Unknown Source)
          at org.apache.derby.impl.sql.compile.FromBaseTable.getStoreCostController(Unknown Source)
          at org.apache.derby.impl.sql.compile.FromBaseTable.estimateCost(Unknown Source)
          at org.apache.derby.impl.sql.compile.OptimizerImpl.estimateTotalCost(Unknown Source)
          at org.apache.derby.impl.sql.compile.OptimizerImpl.costBasedCostOptimizable(Unknown Source)
          at org.apache.derby.impl.sql.compile.OptimizerImpl.costOptimizable(Unknown Source)
          at org.apache.derby.impl.sql.compile.FromBaseTable.optimizeIt(Unknown Source)
          at org.apache.derby.impl.sql.compile.ProjectRestrictNode.optimizeIt(Unknown Source)
          at org.apache.derby.impl.sql.compile.OptimizerImpl.costPermutation(Unknown Source)
          at org.apache.derby.impl.sql.compile.SelectNode.optimize(Unknown Source)
          at org.apache.derby.impl.sql.compile.DMLStatementNode.optimizeStatement(Unknown Source)
          at org.apache.derby.impl.sql.compile.CursorNode.optimizeStatement(Unknown Source)
          at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
          at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source)
          at org.apache.derby.jdbc.Driver30.newEmbedPreparedStatement(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
          at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:442)
          at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:368)
          at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:105)
          at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1561)
          at org.hibernate.loader.Loader.doQuery(Loader.java:661)
          at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:223)
          at org.hibernate.loader.Loader.doList(Loader.java:2147)
          at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2026)
          at org.hibernate.loader.Loader.list(Loader.java:2021)
          at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:109)
          at org.hibernate.impl.SessionImpl.listCustomQuery(SessionImpl.java:1475)
          at org.hibernate.impl.AbstractSessionImpl.list(AbstractSessionImpl.java:121)
          at org.hibernate.impl.SQLQueryImpl.list(SQLQueryImpl.java:168)
          -----------------------------------

          Show
          Bassel Kh added a comment - It is very strange of having these kinds of bugs with such a professional db like Derby. Anyways, the error has changed after using Derby 10.4: 2008-06-11 10:50:37.997 GMT Thread [Worker-60,5,main] (XID = 825230855), (SESSIONID = 29), (DATABASE = db/networkManager), (DRDAID = null), Cleanup action starting 2008-06-11 10:50:37.997 GMT Thread [Worker-60,5,main] (XID = 825230855), (SESSIONID = 29), (DATABASE = db/networkManager), (DRDAID = null), Failed Statement is: select monitor.status as status , count(monitor.status) as statusCount , sum (monitor.metthresholdtimes) as thresholdTimes from NetworkManager.ACTIVE_MONITOR as monitor where monitor.col1 <> 'Yes' group by monitor.status java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method) at org.apache.derby.impl.store.raw.data.AllocPage.ReadContainerInfo(Unknown Source) at org.apache.derby.impl.store.raw.data.FileContainer.readHeader(Unknown Source) at org.apache.derby.impl.store.raw.data.RAFContainer.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.derby.impl.store.raw.data.RAFContainer.openContainer(Unknown Source) at org.apache.derby.impl.store.raw.data.FileContainer.setIdent(Unknown Source) at org.apache.derby.impl.store.raw.data.RAFContainer.setIdentity(Unknown Source) at org.apache.derby.impl.services.cache.ConcurrentCache.find(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(Unknown Source) at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Unknown Source) at org.apache.derby.impl.store.access.btree.index.B2IFactory.readConglomerate(Unknown Source) at org.apache.derby.impl.store.access.RAMAccessManager.conglomCacheFind(Unknown Source) at org.apache.derby.impl.store.access.RAMTransaction.findExistingConglomerate(Unknown Source) at org.apache.derby.impl.store.access.RAMTransaction.openStoreCost(Unknown Source) at org.apache.derby.impl.sql.compile.CompilerContextImpl.getStoreCostController(Unknown Source) at org.apache.derby.impl.sql.compile.FromBaseTable.getStoreCostController(Unknown Source) at org.apache.derby.impl.sql.compile.FromBaseTable.estimateCost(Unknown Source) at org.apache.derby.impl.sql.compile.OptimizerImpl.estimateTotalCost(Unknown Source) at org.apache.derby.impl.sql.compile.OptimizerImpl.costBasedCostOptimizable(Unknown Source) at org.apache.derby.impl.sql.compile.OptimizerImpl.costOptimizable(Unknown Source) at org.apache.derby.impl.sql.compile.FromBaseTable.optimizeIt(Unknown Source) at org.apache.derby.impl.sql.compile.ProjectRestrictNode.optimizeIt(Unknown Source) at org.apache.derby.impl.sql.compile.OptimizerImpl.costPermutation(Unknown Source) at org.apache.derby.impl.sql.compile.SelectNode.optimize(Unknown Source) at org.apache.derby.impl.sql.compile.DMLStatementNode.optimizeStatement(Unknown Source) at org.apache.derby.impl.sql.compile.CursorNode.optimizeStatement(Unknown Source) at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source) at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source) at org.apache.derby.jdbc.Driver30.newEmbedPreparedStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:442) at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:368) at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:105) at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1561) at org.hibernate.loader.Loader.doQuery(Loader.java:661) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:223) at org.hibernate.loader.Loader.doList(Loader.java:2147) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2026) at org.hibernate.loader.Loader.list(Loader.java:2021) at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:109) at org.hibernate.impl.SessionImpl.listCustomQuery(SessionImpl.java:1475) at org.hibernate.impl.AbstractSessionImpl.list(AbstractSessionImpl.java:121) at org.hibernate.impl.SQLQueryImpl.list(SQLQueryImpl.java:168) -----------------------------------
          Hide
          Shahbaz added a comment - - edited

          We migrated to 10.4.1.3, still the same problem but with new exception

          ERROR XSDG3: Meta-data for Container org.apache.derby.impl.store.raw.data.RAFContainer4@1386d2b could not be accessed
          at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
          at org.apache.derby.impl.store.raw.data.RAFContainer.clean(Unknown Source)
          at org.apache.derby.impl.services.cache.ConcurrentCache.cleanAndUnkeepEntry(Unknown Source)

          We are getting the same in embedded mode and DB is newly created.

          Show
          Shahbaz added a comment - - edited We migrated to 10.4.1.3, still the same problem but with new exception ERROR XSDG3: Meta-data for Container org.apache.derby.impl.store.raw.data.RAFContainer4@1386d2b could not be accessed at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.store.raw.data.RAFContainer.clean(Unknown Source) at org.apache.derby.impl.services.cache.ConcurrentCache.cleanAndUnkeepEntry(Unknown Source) We are getting the same in embedded mode and DB is newly created.
          Hide
          Shahbaz added a comment - - edited

          Deby logs with 10.4.1.3

          Show
          Shahbaz added a comment - - edited Deby logs with 10.4.1.3
          Hide
          Mike Matrigali added a comment -

          can you reproduce this in embedded, starting with a new database using the 10.3.3.0 release? This release
          has a fix for a serious problem with I/O access to page 0. It is important to differentiate between testing against
          the new release with a new database and testing against the new release with an existine database. Unfortunately
          once the problem happens no future software version can fix the problem.

          You can see more info about this problem and the announcement of the recent release here:
          http://mail-archives.apache.org/mod_mbox/db-derby-user/200805.mbox/%3c54ac72d70805211242y29d4a0bcyf6ae518a7a8f1767@mail.gmail.com%3e

          Show
          Mike Matrigali added a comment - can you reproduce this in embedded, starting with a new database using the 10.3.3.0 release? This release has a fix for a serious problem with I/O access to page 0. It is important to differentiate between testing against the new release with a new database and testing against the new release with an existine database. Unfortunately once the problem happens no future software version can fix the problem. You can see more info about this problem and the announcement of the recent release here: http://mail-archives.apache.org/mod_mbox/db-derby-user/200805.mbox/%3c54ac72d70805211242y29d4a0bcyf6ae518a7a8f1767@mail.gmail.com%3e
          Hide
          Kristian Waagan added a comment -

          It's good to know that the client driver isn't experiencing the bug. Thanks for testing it!

          Now, to get this fixed in the embedded driver, it would be really helpful to get a repro we can run. Are you still planning on providing one?
          If not, it's very hard to fix the bug, as we don't know the cause. One could guess and try to put some tracing code into a custom build, but that means someone has to do this and then you have to run it in your system to test it out.
          I'm not sure how excited developers will be about such a bug fixing cycle

          Further, I guess we don't know when the corruption is happening, only that you see it when you restart Derby.

          Show
          Kristian Waagan added a comment - It's good to know that the client driver isn't experiencing the bug. Thanks for testing it! Now, to get this fixed in the embedded driver, it would be really helpful to get a repro we can run. Are you still planning on providing one? If not, it's very hard to fix the bug, as we don't know the cause. One could guess and try to put some tracing code into a custom build, but that means someone has to do this and then you have to run it in your system to test it out. I'm not sure how excited developers will be about such a bug fixing cycle Further, I guess we don't know when the corruption is happening, only that you see it when you restart Derby.
          Hide
          Shahbaz added a comment -

          Its not happening with client driver after several testing but we can reproduce the same using embedded mode in just one or two attempts.

          Show
          Shahbaz added a comment - Its not happening with client driver after several testing but we can reproduce the same using embedded mode in just one or two attempts.
          Hide
          Shahbaz added a comment -

          Upto now its working fine with client drvier and still in testing. I also checked our code and we dont have Thread.interrupt() case, It looks to me straight forward. I am sorry s/w is not freely available but let me try to give you enough, so that derby developers can reproduce the same.

          Show
          Shahbaz added a comment - Upto now its working fine with client drvier and still in testing. I also checked our code and we dont have Thread.interrupt() case, It looks to me straight forward. I am sorry s/w is not freely available but let me try to give you enough, so that derby developers can reproduce the same.
          Hide
          Kristian Waagan added a comment -

          That's great Shahbaz.
          I would appreciate if you kept running with the client driver for a while longer to be more certain that the corruption doesn't happen there.

          I can't tell you for sure, but the theory is that something is interrupting the Derby threads when they are writing data to disk.
          In previous cases we have seen lately, this something has been c3p0.

          Note that this is just a theory, and we need to look further into it. Also, I do not know if the latest version of c3p0 has this "problem" or even if it is c3p0 causing the trouble.
          I don't have the overview to tell what needs to be done, but in my opinion it seems Derby is not able to protect itself well enough from these kinds of actions (thread interruption).
          If you have all your source code easily available, you could check if there is any code calling Thread.interrupt().

          Another step, since you seem to be able to reproduce this pretty easy, is to instrument Derby with some debugging code checking if the threads are indeed interrupted.
          Out of curiosity, is the software stack you are using freely available, so Derby developers could run it and provoke the error if they wish to do so?

          Show
          Kristian Waagan added a comment - That's great Shahbaz. I would appreciate if you kept running with the client driver for a while longer to be more certain that the corruption doesn't happen there. I can't tell you for sure, but the theory is that something is interrupting the Derby threads when they are writing data to disk. In previous cases we have seen lately, this something has been c3p0. Note that this is just a theory, and we need to look further into it. Also, I do not know if the latest version of c3p0 has this "problem" or even if it is c3p0 causing the trouble. I don't have the overview to tell what needs to be done, but in my opinion it seems Derby is not able to protect itself well enough from these kinds of actions (thread interruption). If you have all your source code easily available, you could check if there is any code calling Thread.interrupt(). Another step, since you seem to be able to reproduce this pretty easy, is to instrument Derby with some debugging code checking if the threads are indeed interrupted. Out of curiosity, is the software stack you are using freely available, so Derby developers could run it and provoke the error if they wish to do so?
          Hide
          Shahbaz added a comment -

          When I put all our DBs on client driver (network mode), upto now its working fine, but testing is going on. but we will prefer to have in embedded mode. Why its failing in embedded mode?

          Show
          Shahbaz added a comment - When I put all our DBs on client driver (network mode), upto now its working fine, but testing is going on. but we will prefer to have in embedded mode. Why its failing in embedded mode?
          Hide
          Mike Matrigali added a comment -

          I can't tell from the comments. Have you reproduced this problem starting fresh from a new uncorrupted database against the latest 10.4 release or a hand built 10.3 release with the recent I/O bug fixed? Once a checksum error happens it is unlikely any future release will fix it. So if your application has an existing db embedded in it with the error no future software is likely to fix it.

          Show
          Mike Matrigali added a comment - I can't tell from the comments. Have you reproduced this problem starting fresh from a new uncorrupted database against the latest 10.4 release or a hand built 10.3 release with the recent I/O bug fixed? Once a checksum error happens it is unlikely any future release will fix it. So if your application has an existing db embedded in it with the error no future software is likely to fix it.
          Hide
          Kristian Waagan added a comment -

          I see in your Hibernate configuration file that there is a line commented out for using the client driver.
          Have you tried running with the the client driver?

          It would be interesting to see if that fails as well, because that would indicate there is another bug than the thread interruption issue in play.

          Show
          Kristian Waagan added a comment - I see in your Hibernate configuration file that there is a line commented out for using the client driver. Have you tried running with the the client driver? It would be interesting to see if that fails as well, because that would indicate there is another bug than the thread interruption issue in play.
          Hide
          Shahbaz added a comment -

          hibernate cfg files

          Show
          Shahbaz added a comment - hibernate cfg files
          Hide
          Shahbaz added a comment -

          Logs and hibernate cfg files

          Show
          Shahbaz added a comment - Logs and hibernate cfg files
          Hide
          Shahbaz added a comment -

          Its ok if I am using 2 DBs but when I tried for 3rd DB in embedded mode with out c3p0, the same problem appeared again. I am attaching some files for reference.

          Show
          Shahbaz added a comment - Its ok if I am using 2 DBs but when I tried for 3rd DB in embedded mode with out c3p0, the same problem appeared again. I am attaching some files for reference.
          Hide
          Shahbaz added a comment -

          I disbled c3p0, it has reduced the curruption, but it again happened when I tried to restart my service 4-5 times,

          Logs:

          ------------ BEGIN SHUTDOWN ERROR STACK -------------

          ERROR XSDG2: Invalid checksum on Page Page(0,Container(0, 289)), expected=4,121,030,011, on-disk version=3,162,187,389, page dump follows: Hex dump:
          00000000: 0076 0000 0001 0000 0000 0000 0004 0000 .v..............
          00000010: 0000 0006 0000 0000 0000 0000 0000 0000 ................
          00000020: 0000 0000 0001 0000 0000 0000 0000 0000 ................
          00000030: 0000 0000 0000 0000 0000 0000 ffff ffff ................
          00000040: ffff ffff 0000 0000 0000 0000 0000 0000 ................
          00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000060: 0000 0000 0000 0000 0000 0000 5000 0000 ............P...
          00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................

          at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
          at org.apache.derby.impl.store.raw.data.StoredPage.validateChecksum(Unknown Source)
          at org.apache.derby.impl.store.raw.data.StoredPage.initFromData(Unknown Source)
          at org.apache.derby.impl.store.raw.data.AllocPage.initFromData(Unknown Source)
          at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(Unknown Source)
          at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(Unknown Source)
          at org.apache.derby.impl.services.cache.CachedItem.takeOnIdentity(Unknown Source)
          at org.apache.derby.impl.services.cache.Clock.addEntry(Unknown Source)
          at org.apache.derby.impl.services.cache.Clock.find(Unknown Source)
          at org.apache.derby.impl.store.raw.data.FileContainer.getAllocPage(Unknown Source)
          at org.apache.derby.impl.store.raw.data.BaseContainer.getAllocPage(Unknown Source)
          at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getAllocPage(Unknown Source)
          at org.apache.derby.impl.store.raw.data.AllocationCache.validate(Unknown Source)
          at org.apache.derby.impl.store.raw.data.AllocationCache.getLastPageNumber(Unknown Source)
          at org.apache.derby.impl.store.raw.data.FileContainer.pageValid(Unknown Source)
          at org.apache.derby.impl.store.raw.data.FileContainer.getUserPage(Unknown Source)
          at org.apache.derby.impl.store.raw.data.FileContainer.getPage(Unknown Source)
          at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getPage(Unknown Source)
          at org.apache.derby.impl.store.access.btree.ControlRow.get(Unknown Source)
          at org.apache.derby.impl.store.access.btree.index.B2IFactory.readConglomerate(Unknown Source)
          at org.apache.derby.impl.store.access.RAMAccessManager.conglomCacheFind(Unknown Source)
          at org.apache.derby.impl.store.access.RAMTransaction.findExistingConglomerate(Unknown Source)
          at org.apache.derby.impl.store.access.RAMTransaction.openScan(Unknown Source)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getConstraintDescriptorViaIndex(Unknown Source)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getConstraintDescriptorsScan(Unknown Source)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getConstraintDescriptors(Unknown Source)
          at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getConstraintDescriptor(Unknown Source)
          at org.apache.derby.impl.sql.compile.AccessPathImpl.initializeAccessPathName(Unknown Source)
          at org.apache.derby.impl.sql.compile.FromTable.rememberAsBest(Unknown Source)
          at org.apache.derby.impl.sql.compile.ProjectRestrictNode.rememberAsBest(Unknown Source)
          at org.apache.derby.impl.sql.compile.OptimizerImpl.rememberBestCost(Unknown Source)
          at org.apache.derby.impl.sql.compile.OptimizerImpl.getNextDecoratedPermutation(Unknown Source)
          at org.apache.derby.impl.sql.compile.SelectNode.optimize(Unknown Source)
          at org.apache.derby.impl.sql.compile.DMLStatementNode.optimizeStatement(Unknown Source)
          at org.apache.derby.impl.sql.compile.CursorNode.optimizeStatement(Unknown Source)
          at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
          at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
          at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source)
          at org.apache.derby.jdbc.Driver30.newEmbedPreparedStatement(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
          at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:442)
          at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:368)
          at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:105)
          at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1561)
          at org.hibernate.loader.Loader.doQuery(Loader.java:661)
          at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:223)
          at org.hibernate.loader.Loader.doList(Loader.java:2147)
          at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2026)
          at org.hibernate.loader.Loader.list(Loader.java:2021)
          at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:369)
          at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:298)
          at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:137)
          at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1014)
          at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
          at org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:750)
          at ae.sphere.arena.networkManager.db.dao._RootDAO.getCount(_RootDAO.java:130)
          at ae.sphere.arena.configurationManager.db.ConfManagerDBStartup$1.run(ConfManagerDBStartup.java:62)
          at org.eclipse.core.internal.jobs.Worker.run(Worker.java:58)

          Show
          Shahbaz added a comment - I disbled c3p0, it has reduced the curruption, but it again happened when I tried to restart my service 4-5 times, Logs: ------------ BEGIN SHUTDOWN ERROR STACK ------------- ERROR XSDG2: Invalid checksum on Page Page(0,Container(0, 289)), expected=4,121,030,011, on-disk version=3,162,187,389, page dump follows: Hex dump: 00000000: 0076 0000 0001 0000 0000 0000 0004 0000 .v.............. 00000010: 0000 0006 0000 0000 0000 0000 0000 0000 ................ 00000020: 0000 0000 0001 0000 0000 0000 0000 0000 ................ 00000030: 0000 0000 0000 0000 0000 0000 ffff ffff ................ 00000040: ffff ffff 0000 0000 0000 0000 0000 0000 ................ 00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000060: 0000 0000 0000 0000 0000 0000 5000 0000 ............P... 00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................ at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.store.raw.data.StoredPage.validateChecksum(Unknown Source) at org.apache.derby.impl.store.raw.data.StoredPage.initFromData(Unknown Source) at org.apache.derby.impl.store.raw.data.AllocPage.initFromData(Unknown Source) at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(Unknown Source) at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(Unknown Source) at org.apache.derby.impl.services.cache.CachedItem.takeOnIdentity(Unknown Source) at org.apache.derby.impl.services.cache.Clock.addEntry(Unknown Source) at org.apache.derby.impl.services.cache.Clock.find(Unknown Source) at org.apache.derby.impl.store.raw.data.FileContainer.getAllocPage(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseContainer.getAllocPage(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getAllocPage(Unknown Source) at org.apache.derby.impl.store.raw.data.AllocationCache.validate(Unknown Source) at org.apache.derby.impl.store.raw.data.AllocationCache.getLastPageNumber(Unknown Source) at org.apache.derby.impl.store.raw.data.FileContainer.pageValid(Unknown Source) at org.apache.derby.impl.store.raw.data.FileContainer.getUserPage(Unknown Source) at org.apache.derby.impl.store.raw.data.FileContainer.getPage(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getPage(Unknown Source) at org.apache.derby.impl.store.access.btree.ControlRow.get(Unknown Source) at org.apache.derby.impl.store.access.btree.index.B2IFactory.readConglomerate(Unknown Source) at org.apache.derby.impl.store.access.RAMAccessManager.conglomCacheFind(Unknown Source) at org.apache.derby.impl.store.access.RAMTransaction.findExistingConglomerate(Unknown Source) at org.apache.derby.impl.store.access.RAMTransaction.openScan(Unknown Source) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getConstraintDescriptorViaIndex(Unknown Source) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getConstraintDescriptorsScan(Unknown Source) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getConstraintDescriptors(Unknown Source) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getConstraintDescriptor(Unknown Source) at org.apache.derby.impl.sql.compile.AccessPathImpl.initializeAccessPathName(Unknown Source) at org.apache.derby.impl.sql.compile.FromTable.rememberAsBest(Unknown Source) at org.apache.derby.impl.sql.compile.ProjectRestrictNode.rememberAsBest(Unknown Source) at org.apache.derby.impl.sql.compile.OptimizerImpl.rememberBestCost(Unknown Source) at org.apache.derby.impl.sql.compile.OptimizerImpl.getNextDecoratedPermutation(Unknown Source) at org.apache.derby.impl.sql.compile.SelectNode.optimize(Unknown Source) at org.apache.derby.impl.sql.compile.DMLStatementNode.optimizeStatement(Unknown Source) at org.apache.derby.impl.sql.compile.CursorNode.optimizeStatement(Unknown Source) at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source) at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source) at org.apache.derby.jdbc.Driver30.newEmbedPreparedStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:442) at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:368) at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:105) at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1561) at org.hibernate.loader.Loader.doQuery(Loader.java:661) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:223) at org.hibernate.loader.Loader.doList(Loader.java:2147) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2026) at org.hibernate.loader.Loader.list(Loader.java:2021) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:369) at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:298) at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:137) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1014) at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79) at org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:750) at ae.sphere.arena.networkManager.db.dao._RootDAO.getCount(_RootDAO.java:130) at ae.sphere.arena.configurationManager.db.ConfManagerDBStartup$1.run(ConfManagerDBStartup.java:62) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:58)
          Hide
          Kristian Waagan added a comment -

          Okay.
          I see that you are using c3p0, I assume it is for connection pooling?
          Can you try to disable it?

          We know there are some problems using it with Derby, because it may interrupt threads. I'm not sure if that is still the case for the newest version of c3p0.
          Another option is to use the client driver instead of the embedded driver, and see if you still get the corruption.

          Show
          Kristian Waagan added a comment - Okay. I see that you are using c3p0, I assume it is for connection pooling? Can you try to disable it? We know there are some problems using it with Derby, because it may interrupt threads. I'm not sure if that is still the case for the newest version of c3p0. Another option is to use the client driver instead of the embedded driver, and see if you still get the corruption.
          Hide
          Shahbaz added a comment -

          We tried both of the cases, but in actual implementation it is one process, we have tried latest release also but the same result. Its corrupting only some tables.

          Here are some more logs

          Caused by: java.sql.SQLException: Invalid checksum on Page Page(0,Container(0, 1568)), expected=2,264,079,051, on-disk version=6,651,942,472,428,708,205, page dump follows: Hex dump:
          00000000: 0076 0000 0001 0000 0000 0000 0002 0000 .v..............
          00000010: 0000 0006 0000 0000 0000 0000 0000 0000 ................
          00000020: 0000 0000 0001 0000 0000 0000 0000 0000 ................
          00000030: 0000 0000 0000 0000 0000 0000 ffff ffff ................
          00000040: ffff ffff 0000 0000 0000 0000 0000 0000 ................
          00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000060: 0000 0000 0000 0000 0000 0000 5000 0000 ............P...
          00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
          000000c0: 0000 0010 0000 0000 0000 0000 2a73 7973 .............sys
          000000d0: 2d70 6163 6b61 6765 2d6d 6772 2a3a 2070 .package.mgr...p
          000000e0: 726f 6365 7373 696e 6720 6e65 7720 6a61 rocessing.new.ja
          000000f0: 722c 2027 433a 5c50 726f 6772 616d 2046 r...C..Program.F
          00000100: 696c 6573 5c53 7068 6572 6520 4e65 7477 iles.Sphere.Netw
          00000110: 6f72 6b73 5c41 7265 6e61 2050 6c61 7466 orks.Arena.Platf
          00000120: 6f72 6d5c 706c 7567 696e 735c 6165 2e73 orm.plugins.ae.s
          00000130: 7068 6572 652e 6172 656e 612e 6e65 7477 phere.arena.netw
          00000140: 6f72 6b4d 616e 6167 6572 2e63 6f6d 6d6f orkManager.commo
          00000150: 6e5f 322e 322e 315c 636f 6d6d 6f6e 2e6a n.2.2.1.common.j
          00000160: 6172 270d 0a2a 7379 732d 7061 636b 6167 ar....sys.packag
          00000170: 652d 6d67 722a 3a20 7072 6f63 6573 7369 e.mgr...processi
          00000180: 6e67 206e 6577 206a 6172 2c20 2743 3a5c ng.new.jar...C..
          00000190: 5072 6f67 7261 6d20 4669 6c65 735c 5370 Program.Files.Sp
          000001a0: 6865 7265 204e 6574 776f 726b 735c 4172 here.Networks.Ar
          000001b0: 656e 6120 506c 6174 666f 726d 5c70 6c75 ena.Platform.plu
          000001c0: 6769 6e73 5c61 652e 7370 6865 7265 2e61 gins.ae.sphere.a
          000001d0: 7265 6e61 2e73 6572 7665 722e 6576 656e rena.server.even
          000001e0: 744d 616e 6167 6572 5f32 2e32 2e31 5c65 tManager.2.2.1.e
          000001f0: 7665 6e74 4d61 6e61 6765 722e 6a61 7227 ventManager.jar.
          00000200: 0d0a 2a73 7973 2d70 6163 6b61 6765 2d6d ...sys.package.m
          00000210: 6772 2a3a 2070 726f 6365 7373 696e 6720 gr...processing.
          00000220: 6e65 7720 6a61 722c 2027 433a 5c50 726f new.jar...C..Pro
          00000230: 6772 616d 2046 696c 6573 5c53 7068 6572 gram.Files.Spher
          00000240: 6520 4e65 7477 6f72 6b73 5c41 7265 6e61 e.Networks.Arena
          00000250: 2050 6c61 7466 6f72 6d5c 706c 7567 696e .Platform.plugin
          00000260: 735c 6165 2e73 7068 6572 652e 6172 656e s.ae.sphere.aren
          00000270: 612e 7365 7276 6572 2e65 7665 6e74 4d61 a.server.eventMa
          00000280: 6e61 6765 725f 322e 322e 315c 6c69 625c nager.2.2.1.lib.
          00000290: 6163 7469 7661 7469 6f6e 2e6a 6172 270d activation.jar..
          000002a0: 0a2a 7379 732d 7061 636b 6167 652d 6d67 ..sys.package.mg
          000002b0: 722a 3a20 7072 6f63 6573 7369 6e67 206e r...processing.n
          000002c0: 6577 206a 6172 2c20 2743 3a5c 5072 6f67 ew.jar...C..Prog
          000002d0: 7261 6d20 4669 6c65 735c 5370 6865 7265 ram.Files.Sphere
          000002e0: 204e 6574 776f 726b 735c 4172 656e 6120 .Networks.Arena.
          000002f0: 506c 6174 666f 726d 5c70 6c75 6769 6e73 Platform.plugins
          00000300: 5c61 652e 7370 6865 7265 2e61 7265 6e61 .ae.sphere.arena
          00000310: 2e73 6572 7665 722e 6576 656e 744d 616e .server.eventMan
          00000320: 6167 6572 5f32 2e32 2e31 5c6c 6962 5c6a ager.2.2.1.lib.j
          00000330: 616b 6172 7461 2d72 6567 6578 702d 312e akarta.regexp.1.
          00000340: 322e 6a61 7227 0d0a 2a73 7973 2d70 6163 2.jar....sys.pac
          00000350: 6b61 6765 2d6d 6772 2a3a 2070 726f 6365 kage.mgr...proce
          00000360: 7373 696e 6720 6e65 7720 6a61 722c 2027 ssing.new.jar...
          00000370: 433a 5c50 726f 6772 616d 2046 696c 6573 C..Program.Files
          00000380: 5c53 7068 6572 6520 4e65 7477 6f72 6b73 .Sphere.Networks
          00000390: 5c41 7265 6e61 2050 6c61 7466 6f72 6d5c .Arena.Platform.
          000003a0: 706c 7567 696e 735c 6165 2e73 7068 6572 plugins.ae.spher
          000003b0: 652e 6172 656e 612e 7365 7276 6572 2e65 e.arena.server.e
          000003c0: 7665 6e74 4d61 6e61 6765 725f 322e 322e ventManager.2.2.
          000003d0: 315c 6c69 625c 4a43 7570 2e6a 6172 270d 1.lib.JCup.jar..
          000003e0: 0a2a 7379 732d 7061 636b 6167 652d 6d67 ..sys.package.mg
          000003f0: 722a 3a20 7072 6f63 6573 7369 6e67 206e r...processing.n
          00000400: 6577 206a 6172 2c20 2743 3a5c 5072 6f67 ew.jar...C..Prog
          00000410: 7261 6d20 4669 6c65 735c 5370 6865 7265 ram.Files.Sphere
          00000420: 204e 6574 776f 726b 735c 4172 656e 6120 .Networks.Arena.
          00000430: 506c 6174 666f 726d 5c70 6c75 6769 6e73 Platform.plugins
          00000440: 5c61 652e 7370 6865 7265 2e61 7265 6e61 .ae.sphere.arena
          00000450: 2e73 6572 7665 722e 6576 656e 744d 616e .server.eventMan
          00000460: 6167 6572 5f32 2e32 2e31 5c6c 6962 5c6a ager.2.2.1.lib.j
          00000470: 6772 6f75 7073 2d61 6c6c 2e6a 6172 270d groups.all.jar..
          00000480: 0a2a 7379 732d 7061 636b 6167 652d 6d67 ..sys.package.mg
          00000490: 722a 3a20 7072 6f63 6573 7369 6e67 206e r...processing.n
          000004a0: 6577 206a 6172 2c20 2743 3a5c 5072 6f67 ew.jar...C..Prog
          000004b0: 7261 6d20 4669 6c65 735c 5370 6865 7265 ram.Files.Sphere
          000004c0: 204e 6574 776f 726b 735c 4172 656e 6120 .Networks.Arena.
          000004d0: 506c 6174 666f 726d 5c70 6c75 6769 6e73 Platform.plugins
          000004e0: 5c61 652e 7370 6865 7265 2e61 7265 6e61 .ae.sphere.arena
          000004f0: 2e73 6572 7665 722e 6576 656e 744d 616e .server.eventMan
          00000500: 6167 6572 5f32 2e32 2e31 5c6c 6962 5c6a ager.2.2.1.lib.j
          00000510: 6d73 2e6a 6172 270d 0a2a 7379 732d 7061 ms.jar....sys.pa
          00000520: 636b 6167 652d 6d67 722a 3a20 7072 6f63 ckage.mgr...proc
          00000530: 6573 7369 6e67 206e 6577 206a 6172 2c20 essing.new.jar..
          00000540: 2743 3a5c 5072 6f67 7261 6d20 4669 6c65 .C..Program.File
          00000550: 735c 5370 6865 7265 204e 6574 776f 726b s.Sphere.Network
          00000560: 735c 4172 656e 6120 506c 6174 666f 726d s.Arena.Platform
          00000570: 5c70 6c75 6769 6e73 5c61 652e 7370 6865 .plugins.ae.sphe
          00000580: 7265 2e61 7265 6e61 2e73 6572 7665 722e re.arena.server.
          00000590: 6576 656e 744d 616e 6167 6572 5f32 2e32 eventManager.2.2
          000005a0: 2e31 5c6c 6962 5c6a 6d78 7269 2e6a 6172 .1.lib.jmxri.jar
          000005b0: 270d 0a2a 7379 732d 7061 636b 6167 652d ....sys.package.
          000005c0: 6d67 722a 3a20 7072 6f63 6573 7369 6e67 mgr...processing
          000005d0: 206e 6577 206a 6172 2c20 2743 3a5c 5072 .new.jar...C..Pr
          000005e0: 6f67 7261 6d20 4669 6c65 735c 5370 6865 ogram.Files.Sphe
          000005f0: 7265 204e 6574 776f 726b 735c 4172 656e re.Networks.Aren
          00000600: 6120 506c 6174 666f 726d 5c70 6c75 6769 a.Platform.plugi
          00000610: 6e73 5c61 652e 7370 6865 7265 2e61 7265 ns.ae.sphere.are
          00000620: 6e61 2e73 6572 7665 722e 6576 656e 744d na.server.eventM
          00000630: 616e 6167 6572 5f32 2e32 2e31 5c6c 6962 anager.2.2.1.lib
          00000640: 5c6a 6d78 746f 6f6c 732e 6a61 7227 0d0a .jmxtools.jar...
          00000650: 2a73 7973 2d70 6163 6b61 6765 2d6d 6772 .sys.package.mgr
          00000660: 2a3a 2070 726f 6365 7373 696e 6720 6e65 ...processing.ne
          00000670: 7720 6a61 722c 2027 433a 5c50 726f 6772 w.jar...C..Progr
          00000680: 616d 2046 696c 6573 5c53 7068 6572 6520 am.Files.Sphere.
          00000690: 4e65 7477 6f72 6b73 5c41 7265 6e61 2050 Networks.Arena.P
          000006a0: 6c61 7466 6f72 6d5c 706c 7567 696e 735c latform.plugins.
          000006b0: 6165 2e73 7068 6572 652e 6172 656e 612e ae.sphere.arena.
          000006c0: 7365 7276 6572 2e65 7665 6e74 4d61 6e61 server.eventMana
          000006d0: 6765 725f 322e 322e 315c 6c69 625c 6a6e ger.2.2.1.lib.jn
          000006e0: 6469 2e6a 6172 270d 0a2a 7379 732d 7061 di.jar....sys.pa
          000006f0: 636b 6167 652d 6d67 722a 3a20 7072 6f63 ckage.mgr...proc
          00000700: 6573 7369 6e67 206e 6577 206a 6172 2c20 essing.new.jar..
          00000710: 2743 3a5c 5072 6f67 7261 6d20 4669 6c65 .C..Program.File
          00000720: 735c 5370 6865 7265 204e 6574 776f 726b s.Sphere.Network
          00000730: 735c 4172 656e 6120 506c 6174 666f 726d s.Arena.Platform
          00000740: 5c70 6c75 6769 6e73 5c61 652e 7370 6865 .plugins.ae.sphe
          00000750: 7265 2e61 7265 6e61 2e73 6572 7665 722e re.arena.server.
          00000760: 6576 656e 744d 616e 6167 6572 5f32 2e32 eventManager.2.2
          00000770: 2e31 5c6c 6962 5c6a 6f72 616d 2d63 6c69 .1.lib.joram.cli
          00000780: 656e 742e 6a61 7227 0d0a 2a73 7973 2d70 ent.jar....sys.p
          00000790: 6163 6b61 6765 2d6d 6772 2a3a 2070 726f ackage.mgr...pro
          000007a0: 6365 7373 696e 6720 6e65 7720 6a61 722c cessing.new.jar.
          000007b0: 2027 433a 5c50 726f 6772 616d 2046 696c ..C..Program.Fil
          000007c0: 6573 5c53 7068 6572 6520 4e65 7477 6f72 es.Sphere.Networ
          000007d0: 6b73 5c41 7265 6e61 2050 6c61 7466 6f72 ks.Arena.Platfor
          000007e0: 6d5c 706c 7567 696e 735c 6165 2e73 7068 m.plugins.ae.sph
          000007f0: 6572 652e 6172 656e 612e 7365 7276 6572 ere.arena.server
          00000800: 2e65 7665 6e74 4d61 6e61 6765 725f 322e .eventManager.2.
          00000810: 322e 315c 6c69 625c 6a6f 7261 6d2d 6d6f 2.1.lib.joram.mo
          00000820: 6d2e 6a61 7227 0d0a 2a73 7973 2d70 6163 m.jar....sys.pac
          00000830: 6b61 6765 2d6d 6772 2a3a 2070 726f 6365 kage.mgr...proce
          00000840: 7373 696e 6720 6e65 7720 6a61 722c 2027 ssing.new.jar...
          00000850: 433a 5c50 726f 6772 616d 2046 696c 6573 C..Program.Files
          00000860: 5c53 7068 6572 6520 4e65 7477 6f72 6b73 .Sphere.Networks
          00000870: 5c41 7265 6e61 2050 6c61 7466 6f72 6d5c .Arena.Platform.
          00000880: 706c 7567 696e 735c 6165 2e73 7068 6572 plugins.ae.spher
          00000890: 652e 6172 656e 612e 7365 7276 6572 2e65 e.arena.server.e
          000008a0: 7665 6e74 4d61 6e61 6765 725f 322e 322e ventManager.2.2.
          000008b0: 315c 6c69 625c 6a6f 7261 6d2d 7368 6172 1.lib.joram.shar
          000008c0: 6564 2e6a 6172 270d 0a2a 7379 732d 7061 ed.jar....sys.pa
          000008d0: 636b 6167 652d 6d67 722a 3a20 7072 6f63 ckage.mgr...proc
          000008e0: 6573 7369 6e67 206e 6577 206a 6172 2c20 essing.new.jar..
          000008f0: 2743 3a5c 5072 6f67 7261 6d20 4669 6c65 .C..Program.File
          00000900: 735c 5370 6865 7265 204e 6574 776f 726b s.Sphere.Network
          00000910: 735c 4172 656e 6120 506c 6174 666f 726d s.Arena.Platform
          00000920: 5c70 6c75 6769 6e73 5c61 652e 7370 6865 .plugins.ae.sphe
          00000930: 7265 2e61 7265 6e61 2e73 6572 7665 722e re.arena.server.
          00000940: 6576 656e 744d 616e 6167 6572 5f32 2e32 eventManager.2.2
          00000950: 2e31 5c6c 6962 5c6a 7461 2e6a 6172 270d .1.lib.jta.jar..
          00000960: 0a2a 7379 732d 7061 636b 6167 652d 6d67 ..sys.package.mg
          00000970: 722a 3a20 7072 6f63 6573 7369 6e67 206e r...processing.n
          00000980: 6577 206a 6172 2c20 2743 3a5c 5072 6f67 ew.jar...C..Prog
          00000990: 7261 6d20 4669 6c65 735c 5370 6865 7265 ram.Files.Sphere
          000009a0: 204e 6574 776f 726b 735c 4172 656e 6120 .Networks.Arena.
          000009b0: 506c 6174 666f 726d 5c70 6c75 6769 6e73 Platform.plugins
          000009c0: 5c61 652e 7370 6865 7265 2e61 7265 6e61 .ae.sphere.arena
          000009d0: 2e73 6572 7665 722e 6576 656e 744d 616e .server.eventMan
          000009e0: 6167 6572 5f32 2e32 2e31 5c6c 6962 5c6b ager.2.2.1.lib.k
          000009f0: 786d 6c2e 6a61 7227 0d0a 2a73 7973 2d70 xml.jar....sys.p
          00000a00: 6163 6b61 6765 2d6d 6772 2a3a 2070 726f ackage.mgr...pro
          00000a10: 6365 7373 696e 6720 6e65 7720 6a61 722c cessing.new.jar.
          00000a20: 2027 433a 5c50 726f 6772 616d 2046 696c ..C..Program.Fil
          00000a30: 6573 5c53 7068 6572 6520 4e65 7477 6f72 es.Sphere.Networ
          00000a40: 6b73 5c41 7265 6e61 2050 6c61 7466 6f72 ks.Arena.Platfor
          00000a50: 6d5c 706c 7567 696e 735c 6165 2e73 7068 m.plugins.ae.sph
          00000a60: 6572 652e 6172 656e 612e 7365 7276 6572 ere.arena.server
          00000a70: 2e65 7665 6e74 4d61 6e61 6765 725f 322e .eventManager.2.
          00000a80: 322e 315c 6c69 625c 6d69 6470 6170 692e 2.1.lib.midpapi.
          00000a90: 6a61 7227 0d0a 2a73 7973 2d70 6163 6b61 jar....sys.packa
          00000aa0: 6765 2d6d 6772 2a3a 2070 726f 6365 7373 ge.mgr...process
          00000ab0: 696e 6720 6e65 7720 6a61 722c 2027 433a ing.new.jar...C.
          00000ac0: 5c50 726f 6772 616d 2046 696c 6573 5c53 .Program.Files.S
          00000ad0: 7068 6572 6520 4e65 7477 6f72 6b73 5c41 phere.Networks.A
          00000ae0: 7265 6e61 2050 6c61 7466 6f72 6d5c 706c rena.Platform.pl
          00000af0: 7567 696e 735c 6165 2e73 7068 6572 652e ugins.ae.sphere.
          00000b00: 6172 656e 612e 7365 7276 6572 2e65 7665 arena.server.eve
          00000b10: 6e74 4d61 6e61 6765 725f 322e 322e 315c ntManager.2.2.1.
          00000b20: 6c69 625c 6f77 5f6d 6f6e 6f6c 6f67 2e6a lib.ow.monolog.j
          00000b30: 6172 270d 0a2a 7379 732d 7061 636b 6167 ar....sys.packag
          00000b40: 652d 6d67 722a 3a20 7072 6f63 6573 7369 e.mgr...processi
          00000b50: 6e67 206e 6577 206a 6172 2c20 2743 3a5c ng.new.jar...C..
          00000b60: 5072 6f67 7261 6d20 4669 6c65 735c 5370 Program.Files.Sp
          00000b70: 6865 7265 204e 6574 776f 726b 735c 4172 here.Networks.Ar
          00000b80: 656e 6120 506c 6174 666f 726d 5c70 6c75 ena.Platform.plu
          00000b90: 6769 6e73 5c61 652e 7370 6865 7265 2e61 gins.ae.sphere.a
          00000ba0: 7265 6e61 2e73 6572 7665 722e 6576 656e rena.server.even
          00000bb0: 744d 616e 6167 6572 5f32 2e32 2e31 5c6c tManager.2.2.1.l
          00000bc0: 6962 5c73 6f61 702e 6a61 7227 0d0a 2a73 ib.soap.jar....s
          00000bd0: 7973 2d70 6163 6b61 6765 2d6d 6772 2a3a ys.package.mgr..
          00000be0: 2070 726f 6365 7373 696e 6720 6e65 7720 .processing.new.
          00000bf0: 6a61 722c 2027 433a 5c50 726f 6772 616d jar...C..Program
          00000c00: 2046 696c 6573 5c53 7068 6572 6520 4e65 .Files.Sphere.Ne
          00000c10: 7477 6f72 6b73 5c41 7265 6e61 2050 6c61 tworks.Arena.Pla
          00000c20: 7466 6f72 6d5c 706c 7567 696e 735c 6165 tform.plugins.ae
          00000c30: 2e73 7068 6572 652e 6172 656e 612e 6c6f .sphere.arena.lo
          00000c40: 675f 322e 322e 315c 736c 6f67 2e6a 6172 g.2.2.1.slog.jar
          00000c50: 270d 0a2a 7379 732d 7061 636b 6167 652d ....sys.package.
          00000c60: 6d67 722a 3a20 7072 6f63 6573 7369 6e67 mgr...processing
          00000c70: 206e 6577 206a 6172 2c20 2743 3a5c 5072 .new.jar...C..Pr
          00000c80: 6f67 7261 6d20 4669 6c65 735c 5370 6865 ogram.Files.Sphe
          00000c90: 7265 204e 6574 776f 726b 735c 4172 656e re.Networks.Aren
          00000ca0: 6120 506c 6174 666f 726d 5c70 6c75 6769 a.Platform.plugi
          00000cb0: 6e73 5c61 652e 7370 6865 7265 2e61 7265 ns.ae.sphere.are
          00000cc0: 6e61 2e6c 6f67 5f32 2e32 2e31 5c6c 6962 na.log.2.2.1.lib
          00000cd0: 5c63 6f6d 6d6f 6e73 2d6c 6f67 6769 6e67 .commons.logging
          00000ce0: 2d31 2e30 2e34 2e6a 6172 270d 0a2a 7379 .1.0.4.jar....sy
          00000cf0: 732d 7061 636b 6167 652d 6d67 722a 3a20 s.package.mgr...
          00000d00: 7072 6f63 6573 7369 6e67 206e 6577 206a processing.new.j
          00000d10: 6172 2c20 2743 3a5c 5072 6f67 7261 6d20 ar...C..Program.
          00000d20: 4669 6c65 735c 5370 6865 7265 204e 6574 Files.Sphere.Net
          00000d30: 776f 726b 735c 4172 656e 6120 506c 6174 works.Arena.Plat
          00000d40: 666f 726d 5c70 6c75 6769 6e73 5c61 652e form.plugins.ae.
          00000d50: 7370 6865 7265 2e61 7265 6e61 2e6c 6f67 sphere.arena.log
          00000d60: 5f32 2e32 2e31 5c6c 6962 5c6c 6f67 346a .2.2.1.lib.log4j
          00000d70: 2d31 2e32 2e31 322e 6a61 7227 0d0a 2a73 .1.2.12.jar....s
          00000d80: 7973 2d70 6163 6b61 6765 2d6d 6772 2a3a ys.package.mgr..
          00000d90: 2070 726f 6365 7373 696e 6720 6e65 7720 .processing.new.
          00000da0: 6a61 722c 2027 433a 5c50 726f 6772 616d jar...C..Program
          00000db0: 2046 696c 6573 5c53 7068 6572 6520 4e65 .Files.Sphere.Ne
          00000dc0: 7477 6f72 6b73 5c41 7265 6e61 2050 6c61 tworks.Arena.Pla
          00000dd0: 7466 6f72 6d5c 706c 7567 696e 735c 6165 tform.plugins.ae
          00000de0: 2e73 7068 6572 652e 6172 656e 612e 6e65 .sphere.arena.ne
          00000df0: 7477 6f72 6b4d 616e 6167 6572 2e64 6973 tworkManager.dis
          00000e00: 636f 7665 7279 5f32 2e33 2e30 5c64 6973 covery.2.3.0.dis
          00000e10: 636f 7665 7279 2e6a 6172 270d 0a2a 7379 covery.jar....sy
          00000e20: 732d 7061 636b 6167 652d 6d67 722a 3a20 s.package.mgr...
          00000e30: 7072 6f63 6573 7369 6e67 206e 6577 206a processing.new.j
          00000e40: 6172 2c20 2743 3a5c 5072 6f67 7261 6d20 ar...C..Program.
          00000e50: 4669 6c65 735c 5370 6865 7265 204e 6574 Files.Sphere.Net
          00000e60: 776f 726b 735c 4172 656e 6120 506c 6174 works.Arena.Plat
          00000e70: 666f 726d 5c70 6c75 6769 6e73 5c61 652e form.plugins.ae.
          00000e80: 7370 6865 7265 2e61 7265 6e61 2e73 6572 sphere.arena.ser
          00000e90: 7665 725f 322e 322e 315c 6172 656e 6153 ver.2.2.1.arenaS
          00000ea0: 6572 7665 722e 6a61 7227 0d0a 2a73 7973 erver.jar....sys
          00000eb0: 2d70 6163 6b61 6765 2d6d 6772 2a3a 2070 .package.mgr...p
          00000ec0: 726f 6365 7373 696e 6720 6e65 7720 6a61 rocessing.new.ja
          00000ed0: 722c 2027 433a 5c50 726f 6772 616d 2046 r...C..Program.F
          00000ee0: 696c 6573 5c53 7068 6572 6520 4e65 7477 iles.Sphere.Netw
          00000ef0: 6f72 6b73 5c41 7265 6e61 2050 6c61 7466 orks.Arena.Platf
          00000f00: 6f72 6d5c 706c 7567 696e 735c 6165 2e73 orm.plugins.ae.s
          00000f10: 7068 6572 652e 6172 656e 612e 636f 6d6d phere.arena.comm
          00000f20: 6f6e 5f32 2e32 2e31 5c63 6f6d 6d6f 6e2e on.2.2.1.common.
          00000f30: 6a61 7227 0d0a 2a73 7973 2d70 6163 6b61 jar....sys.packa
          00000f40: 6765 2d6d 6772 2a3a 2070 726f 6365 7373 ge.mgr...process
          00000f50: 696e 6720 6e65 7720 6a61 722c 2027 433a ing.new.jar...C.
          00000f60: 5c50 726f 6772 616d 2046 696c 6573 5c53 .Program.Files.S
          00000f70: 7068 6572 6520 4e65 7477 6f72 6b73 5c41 phere.Networks.A
          00000f80: 7265 6e61 2050 6c61 7466 6f72 6d5c 706c rena.Platform.pl
          00000f90: 7567 696e 735c 6165 2e73 7068 6572 652e ugins.ae.sphere.
          00000fa0: 6172 656e 612e 636f 6d6d 6f6e 5f32 2e32 arena.common.2.2
          00000fb0: 2e31 5c6c 6962 5c63 6f6d 6d6f 6e73 2d6c .1.lib.commons.l
          00000fc0: 616e 672d 322e 332e 6a61 7227 0d0a 2a73 ang.2.3.jar....s
          00000fd0: 7973 2d70 6163 6b61 6765 2d6d 6772 2a3a ys.package.mgr..
          00000fe0: 2070 726f 6365 7373 696e 6720 6e65 7720 .processing.new.
          00000ff0: 6a61 722c 2027 433a 5c50 726f 6772 616d jar...C..Program

          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
          at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
          at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source)
          at org.apache.derby.jdbc.Driver30.newEmbedPreparedStatement(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
          at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask.run(GooGooStatementCache.java:525)
          at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

          Show
          Shahbaz added a comment - We tried both of the cases, but in actual implementation it is one process, we have tried latest release also but the same result. Its corrupting only some tables. Here are some more logs Caused by: java.sql.SQLException: Invalid checksum on Page Page(0,Container(0, 1568)), expected=2,264,079,051, on-disk version=6,651,942,472,428,708,205, page dump follows: Hex dump: 00000000: 0076 0000 0001 0000 0000 0000 0002 0000 .v.............. 00000010: 0000 0006 0000 0000 0000 0000 0000 0000 ................ 00000020: 0000 0000 0001 0000 0000 0000 0000 0000 ................ 00000030: 0000 0000 0000 0000 0000 0000 ffff ffff ................ 00000040: ffff ffff 0000 0000 0000 0000 0000 0000 ................ 00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000060: 0000 0000 0000 0000 0000 0000 5000 0000 ............P... 00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000000c0: 0000 0010 0000 0000 0000 0000 2a73 7973 .............sys 000000d0: 2d70 6163 6b61 6765 2d6d 6772 2a3a 2070 .package.mgr...p 000000e0: 726f 6365 7373 696e 6720 6e65 7720 6a61 rocessing.new.ja 000000f0: 722c 2027 433a 5c50 726f 6772 616d 2046 r...C..Program.F 00000100: 696c 6573 5c53 7068 6572 6520 4e65 7477 iles.Sphere.Netw 00000110: 6f72 6b73 5c41 7265 6e61 2050 6c61 7466 orks.Arena.Platf 00000120: 6f72 6d5c 706c 7567 696e 735c 6165 2e73 orm.plugins.ae.s 00000130: 7068 6572 652e 6172 656e 612e 6e65 7477 phere.arena.netw 00000140: 6f72 6b4d 616e 6167 6572 2e63 6f6d 6d6f orkManager.commo 00000150: 6e5f 322e 322e 315c 636f 6d6d 6f6e 2e6a n.2.2.1.common.j 00000160: 6172 270d 0a2a 7379 732d 7061 636b 6167 ar....sys.packag 00000170: 652d 6d67 722a 3a20 7072 6f63 6573 7369 e.mgr...processi 00000180: 6e67 206e 6577 206a 6172 2c20 2743 3a5c ng.new.jar...C.. 00000190: 5072 6f67 7261 6d20 4669 6c65 735c 5370 Program.Files.Sp 000001a0: 6865 7265 204e 6574 776f 726b 735c 4172 here.Networks.Ar 000001b0: 656e 6120 506c 6174 666f 726d 5c70 6c75 ena.Platform.plu 000001c0: 6769 6e73 5c61 652e 7370 6865 7265 2e61 gins.ae.sphere.a 000001d0: 7265 6e61 2e73 6572 7665 722e 6576 656e rena.server.even 000001e0: 744d 616e 6167 6572 5f32 2e32 2e31 5c65 tManager.2.2.1.e 000001f0: 7665 6e74 4d61 6e61 6765 722e 6a61 7227 ventManager.jar. 00000200: 0d0a 2a73 7973 2d70 6163 6b61 6765 2d6d ...sys.package.m 00000210: 6772 2a3a 2070 726f 6365 7373 696e 6720 gr...processing. 00000220: 6e65 7720 6a61 722c 2027 433a 5c50 726f new.jar...C..Pro 00000230: 6772 616d 2046 696c 6573 5c53 7068 6572 gram.Files.Spher 00000240: 6520 4e65 7477 6f72 6b73 5c41 7265 6e61 e.Networks.Arena 00000250: 2050 6c61 7466 6f72 6d5c 706c 7567 696e .Platform.plugin 00000260: 735c 6165 2e73 7068 6572 652e 6172 656e s.ae.sphere.aren 00000270: 612e 7365 7276 6572 2e65 7665 6e74 4d61 a.server.eventMa 00000280: 6e61 6765 725f 322e 322e 315c 6c69 625c nager.2.2.1.lib. 00000290: 6163 7469 7661 7469 6f6e 2e6a 6172 270d activation.jar.. 000002a0: 0a2a 7379 732d 7061 636b 6167 652d 6d67 ..sys.package.mg 000002b0: 722a 3a20 7072 6f63 6573 7369 6e67 206e r...processing.n 000002c0: 6577 206a 6172 2c20 2743 3a5c 5072 6f67 ew.jar...C..Prog 000002d0: 7261 6d20 4669 6c65 735c 5370 6865 7265 ram.Files.Sphere 000002e0: 204e 6574 776f 726b 735c 4172 656e 6120 .Networks.Arena. 000002f0: 506c 6174 666f 726d 5c70 6c75 6769 6e73 Platform.plugins 00000300: 5c61 652e 7370 6865 7265 2e61 7265 6e61 .ae.sphere.arena 00000310: 2e73 6572 7665 722e 6576 656e 744d 616e .server.eventMan 00000320: 6167 6572 5f32 2e32 2e31 5c6c 6962 5c6a ager.2.2.1.lib.j 00000330: 616b 6172 7461 2d72 6567 6578 702d 312e akarta.regexp.1. 00000340: 322e 6a61 7227 0d0a 2a73 7973 2d70 6163 2.jar....sys.pac 00000350: 6b61 6765 2d6d 6772 2a3a 2070 726f 6365 kage.mgr...proce 00000360: 7373 696e 6720 6e65 7720 6a61 722c 2027 ssing.new.jar... 00000370: 433a 5c50 726f 6772 616d 2046 696c 6573 C..Program.Files 00000380: 5c53 7068 6572 6520 4e65 7477 6f72 6b73 .Sphere.Networks 00000390: 5c41 7265 6e61 2050 6c61 7466 6f72 6d5c .Arena.Platform. 000003a0: 706c 7567 696e 735c 6165 2e73 7068 6572 plugins.ae.spher 000003b0: 652e 6172 656e 612e 7365 7276 6572 2e65 e.arena.server.e 000003c0: 7665 6e74 4d61 6e61 6765 725f 322e 322e ventManager.2.2. 000003d0: 315c 6c69 625c 4a43 7570 2e6a 6172 270d 1.lib.JCup.jar.. 000003e0: 0a2a 7379 732d 7061 636b 6167 652d 6d67 ..sys.package.mg 000003f0: 722a 3a20 7072 6f63 6573 7369 6e67 206e r...processing.n 00000400: 6577 206a 6172 2c20 2743 3a5c 5072 6f67 ew.jar...C..Prog 00000410: 7261 6d20 4669 6c65 735c 5370 6865 7265 ram.Files.Sphere 00000420: 204e 6574 776f 726b 735c 4172 656e 6120 .Networks.Arena. 00000430: 506c 6174 666f 726d 5c70 6c75 6769 6e73 Platform.plugins 00000440: 5c61 652e 7370 6865 7265 2e61 7265 6e61 .ae.sphere.arena 00000450: 2e73 6572 7665 722e 6576 656e 744d 616e .server.eventMan 00000460: 6167 6572 5f32 2e32 2e31 5c6c 6962 5c6a ager.2.2.1.lib.j 00000470: 6772 6f75 7073 2d61 6c6c 2e6a 6172 270d groups.all.jar.. 00000480: 0a2a 7379 732d 7061 636b 6167 652d 6d67 ..sys.package.mg 00000490: 722a 3a20 7072 6f63 6573 7369 6e67 206e r...processing.n 000004a0: 6577 206a 6172 2c20 2743 3a5c 5072 6f67 ew.jar...C..Prog 000004b0: 7261 6d20 4669 6c65 735c 5370 6865 7265 ram.Files.Sphere 000004c0: 204e 6574 776f 726b 735c 4172 656e 6120 .Networks.Arena. 000004d0: 506c 6174 666f 726d 5c70 6c75 6769 6e73 Platform.plugins 000004e0: 5c61 652e 7370 6865 7265 2e61 7265 6e61 .ae.sphere.arena 000004f0: 2e73 6572 7665 722e 6576 656e 744d 616e .server.eventMan 00000500: 6167 6572 5f32 2e32 2e31 5c6c 6962 5c6a ager.2.2.1.lib.j 00000510: 6d73 2e6a 6172 270d 0a2a 7379 732d 7061 ms.jar....sys.pa 00000520: 636b 6167 652d 6d67 722a 3a20 7072 6f63 ckage.mgr...proc 00000530: 6573 7369 6e67 206e 6577 206a 6172 2c20 essing.new.jar.. 00000540: 2743 3a5c 5072 6f67 7261 6d20 4669 6c65 .C..Program.File 00000550: 735c 5370 6865 7265 204e 6574 776f 726b s.Sphere.Network 00000560: 735c 4172 656e 6120 506c 6174 666f 726d s.Arena.Platform 00000570: 5c70 6c75 6769 6e73 5c61 652e 7370 6865 .plugins.ae.sphe 00000580: 7265 2e61 7265 6e61 2e73 6572 7665 722e re.arena.server. 00000590: 6576 656e 744d 616e 6167 6572 5f32 2e32 eventManager.2.2 000005a0: 2e31 5c6c 6962 5c6a 6d78 7269 2e6a 6172 .1.lib.jmxri.jar 000005b0: 270d 0a2a 7379 732d 7061 636b 6167 652d ....sys.package. 000005c0: 6d67 722a 3a20 7072 6f63 6573 7369 6e67 mgr...processing 000005d0: 206e 6577 206a 6172 2c20 2743 3a5c 5072 .new.jar...C..Pr 000005e0: 6f67 7261 6d20 4669 6c65 735c 5370 6865 ogram.Files.Sphe 000005f0: 7265 204e 6574 776f 726b 735c 4172 656e re.Networks.Aren 00000600: 6120 506c 6174 666f 726d 5c70 6c75 6769 a.Platform.plugi 00000610: 6e73 5c61 652e 7370 6865 7265 2e61 7265 ns.ae.sphere.are 00000620: 6e61 2e73 6572 7665 722e 6576 656e 744d na.server.eventM 00000630: 616e 6167 6572 5f32 2e32 2e31 5c6c 6962 anager.2.2.1.lib 00000640: 5c6a 6d78 746f 6f6c 732e 6a61 7227 0d0a .jmxtools.jar... 00000650: 2a73 7973 2d70 6163 6b61 6765 2d6d 6772 .sys.package.mgr 00000660: 2a3a 2070 726f 6365 7373 696e 6720 6e65 ...processing.ne 00000670: 7720 6a61 722c 2027 433a 5c50 726f 6772 w.jar...C..Progr 00000680: 616d 2046 696c 6573 5c53 7068 6572 6520 am.Files.Sphere. 00000690: 4e65 7477 6f72 6b73 5c41 7265 6e61 2050 Networks.Arena.P 000006a0: 6c61 7466 6f72 6d5c 706c 7567 696e 735c latform.plugins. 000006b0: 6165 2e73 7068 6572 652e 6172 656e 612e ae.sphere.arena. 000006c0: 7365 7276 6572 2e65 7665 6e74 4d61 6e61 server.eventMana 000006d0: 6765 725f 322e 322e 315c 6c69 625c 6a6e ger.2.2.1.lib.jn 000006e0: 6469 2e6a 6172 270d 0a2a 7379 732d 7061 di.jar....sys.pa 000006f0: 636b 6167 652d 6d67 722a 3a20 7072 6f63 ckage.mgr...proc 00000700: 6573 7369 6e67 206e 6577 206a 6172 2c20 essing.new.jar.. 00000710: 2743 3a5c 5072 6f67 7261 6d20 4669 6c65 .C..Program.File 00000720: 735c 5370 6865 7265 204e 6574 776f 726b s.Sphere.Network 00000730: 735c 4172 656e 6120 506c 6174 666f 726d s.Arena.Platform 00000740: 5c70 6c75 6769 6e73 5c61 652e 7370 6865 .plugins.ae.sphe 00000750: 7265 2e61 7265 6e61 2e73 6572 7665 722e re.arena.server. 00000760: 6576 656e 744d 616e 6167 6572 5f32 2e32 eventManager.2.2 00000770: 2e31 5c6c 6962 5c6a 6f72 616d 2d63 6c69 .1.lib.joram.cli 00000780: 656e 742e 6a61 7227 0d0a 2a73 7973 2d70 ent.jar....sys.p 00000790: 6163 6b61 6765 2d6d 6772 2a3a 2070 726f ackage.mgr...pro 000007a0: 6365 7373 696e 6720 6e65 7720 6a61 722c cessing.new.jar. 000007b0: 2027 433a 5c50 726f 6772 616d 2046 696c ..C..Program.Fil 000007c0: 6573 5c53 7068 6572 6520 4e65 7477 6f72 es.Sphere.Networ 000007d0: 6b73 5c41 7265 6e61 2050 6c61 7466 6f72 ks.Arena.Platfor 000007e0: 6d5c 706c 7567 696e 735c 6165 2e73 7068 m.plugins.ae.sph 000007f0: 6572 652e 6172 656e 612e 7365 7276 6572 ere.arena.server 00000800: 2e65 7665 6e74 4d61 6e61 6765 725f 322e .eventManager.2. 00000810: 322e 315c 6c69 625c 6a6f 7261 6d2d 6d6f 2.1.lib.joram.mo 00000820: 6d2e 6a61 7227 0d0a 2a73 7973 2d70 6163 m.jar....sys.pac 00000830: 6b61 6765 2d6d 6772 2a3a 2070 726f 6365 kage.mgr...proce 00000840: 7373 696e 6720 6e65 7720 6a61 722c 2027 ssing.new.jar... 00000850: 433a 5c50 726f 6772 616d 2046 696c 6573 C..Program.Files 00000860: 5c53 7068 6572 6520 4e65 7477 6f72 6b73 .Sphere.Networks 00000870: 5c41 7265 6e61 2050 6c61 7466 6f72 6d5c .Arena.Platform. 00000880: 706c 7567 696e 735c 6165 2e73 7068 6572 plugins.ae.spher 00000890: 652e 6172 656e 612e 7365 7276 6572 2e65 e.arena.server.e 000008a0: 7665 6e74 4d61 6e61 6765 725f 322e 322e ventManager.2.2. 000008b0: 315c 6c69 625c 6a6f 7261 6d2d 7368 6172 1.lib.joram.shar 000008c0: 6564 2e6a 6172 270d 0a2a 7379 732d 7061 ed.jar....sys.pa 000008d0: 636b 6167 652d 6d67 722a 3a20 7072 6f63 ckage.mgr...proc 000008e0: 6573 7369 6e67 206e 6577 206a 6172 2c20 essing.new.jar.. 000008f0: 2743 3a5c 5072 6f67 7261 6d20 4669 6c65 .C..Program.File 00000900: 735c 5370 6865 7265 204e 6574 776f 726b s.Sphere.Network 00000910: 735c 4172 656e 6120 506c 6174 666f 726d s.Arena.Platform 00000920: 5c70 6c75 6769 6e73 5c61 652e 7370 6865 .plugins.ae.sphe 00000930: 7265 2e61 7265 6e61 2e73 6572 7665 722e re.arena.server. 00000940: 6576 656e 744d 616e 6167 6572 5f32 2e32 eventManager.2.2 00000950: 2e31 5c6c 6962 5c6a 7461 2e6a 6172 270d .1.lib.jta.jar.. 00000960: 0a2a 7379 732d 7061 636b 6167 652d 6d67 ..sys.package.mg 00000970: 722a 3a20 7072 6f63 6573 7369 6e67 206e r...processing.n 00000980: 6577 206a 6172 2c20 2743 3a5c 5072 6f67 ew.jar...C..Prog 00000990: 7261 6d20 4669 6c65 735c 5370 6865 7265 ram.Files.Sphere 000009a0: 204e 6574 776f 726b 735c 4172 656e 6120 .Networks.Arena. 000009b0: 506c 6174 666f 726d 5c70 6c75 6769 6e73 Platform.plugins 000009c0: 5c61 652e 7370 6865 7265 2e61 7265 6e61 .ae.sphere.arena 000009d0: 2e73 6572 7665 722e 6576 656e 744d 616e .server.eventMan 000009e0: 6167 6572 5f32 2e32 2e31 5c6c 6962 5c6b ager.2.2.1.lib.k 000009f0: 786d 6c2e 6a61 7227 0d0a 2a73 7973 2d70 xml.jar....sys.p 00000a00: 6163 6b61 6765 2d6d 6772 2a3a 2070 726f ackage.mgr...pro 00000a10: 6365 7373 696e 6720 6e65 7720 6a61 722c cessing.new.jar. 00000a20: 2027 433a 5c50 726f 6772 616d 2046 696c ..C..Program.Fil 00000a30: 6573 5c53 7068 6572 6520 4e65 7477 6f72 es.Sphere.Networ 00000a40: 6b73 5c41 7265 6e61 2050 6c61 7466 6f72 ks.Arena.Platfor 00000a50: 6d5c 706c 7567 696e 735c 6165 2e73 7068 m.plugins.ae.sph 00000a60: 6572 652e 6172 656e 612e 7365 7276 6572 ere.arena.server 00000a70: 2e65 7665 6e74 4d61 6e61 6765 725f 322e .eventManager.2. 00000a80: 322e 315c 6c69 625c 6d69 6470 6170 692e 2.1.lib.midpapi. 00000a90: 6a61 7227 0d0a 2a73 7973 2d70 6163 6b61 jar....sys.packa 00000aa0: 6765 2d6d 6772 2a3a 2070 726f 6365 7373 ge.mgr...process 00000ab0: 696e 6720 6e65 7720 6a61 722c 2027 433a ing.new.jar...C. 00000ac0: 5c50 726f 6772 616d 2046 696c 6573 5c53 .Program.Files.S 00000ad0: 7068 6572 6520 4e65 7477 6f72 6b73 5c41 phere.Networks.A 00000ae0: 7265 6e61 2050 6c61 7466 6f72 6d5c 706c rena.Platform.pl 00000af0: 7567 696e 735c 6165 2e73 7068 6572 652e ugins.ae.sphere. 00000b00: 6172 656e 612e 7365 7276 6572 2e65 7665 arena.server.eve 00000b10: 6e74 4d61 6e61 6765 725f 322e 322e 315c ntManager.2.2.1. 00000b20: 6c69 625c 6f77 5f6d 6f6e 6f6c 6f67 2e6a lib.ow.monolog.j 00000b30: 6172 270d 0a2a 7379 732d 7061 636b 6167 ar....sys.packag 00000b40: 652d 6d67 722a 3a20 7072 6f63 6573 7369 e.mgr...processi 00000b50: 6e67 206e 6577 206a 6172 2c20 2743 3a5c ng.new.jar...C.. 00000b60: 5072 6f67 7261 6d20 4669 6c65 735c 5370 Program.Files.Sp 00000b70: 6865 7265 204e 6574 776f 726b 735c 4172 here.Networks.Ar 00000b80: 656e 6120 506c 6174 666f 726d 5c70 6c75 ena.Platform.plu 00000b90: 6769 6e73 5c61 652e 7370 6865 7265 2e61 gins.ae.sphere.a 00000ba0: 7265 6e61 2e73 6572 7665 722e 6576 656e rena.server.even 00000bb0: 744d 616e 6167 6572 5f32 2e32 2e31 5c6c tManager.2.2.1.l 00000bc0: 6962 5c73 6f61 702e 6a61 7227 0d0a 2a73 ib.soap.jar....s 00000bd0: 7973 2d70 6163 6b61 6765 2d6d 6772 2a3a ys.package.mgr.. 00000be0: 2070 726f 6365 7373 696e 6720 6e65 7720 .processing.new. 00000bf0: 6a61 722c 2027 433a 5c50 726f 6772 616d jar...C..Program 00000c00: 2046 696c 6573 5c53 7068 6572 6520 4e65 .Files.Sphere.Ne 00000c10: 7477 6f72 6b73 5c41 7265 6e61 2050 6c61 tworks.Arena.Pla 00000c20: 7466 6f72 6d5c 706c 7567 696e 735c 6165 tform.plugins.ae 00000c30: 2e73 7068 6572 652e 6172 656e 612e 6c6f .sphere.arena.lo 00000c40: 675f 322e 322e 315c 736c 6f67 2e6a 6172 g.2.2.1.slog.jar 00000c50: 270d 0a2a 7379 732d 7061 636b 6167 652d ....sys.package. 00000c60: 6d67 722a 3a20 7072 6f63 6573 7369 6e67 mgr...processing 00000c70: 206e 6577 206a 6172 2c20 2743 3a5c 5072 .new.jar...C..Pr 00000c80: 6f67 7261 6d20 4669 6c65 735c 5370 6865 ogram.Files.Sphe 00000c90: 7265 204e 6574 776f 726b 735c 4172 656e re.Networks.Aren 00000ca0: 6120 506c 6174 666f 726d 5c70 6c75 6769 a.Platform.plugi 00000cb0: 6e73 5c61 652e 7370 6865 7265 2e61 7265 ns.ae.sphere.are 00000cc0: 6e61 2e6c 6f67 5f32 2e32 2e31 5c6c 6962 na.log.2.2.1.lib 00000cd0: 5c63 6f6d 6d6f 6e73 2d6c 6f67 6769 6e67 .commons.logging 00000ce0: 2d31 2e30 2e34 2e6a 6172 270d 0a2a 7379 .1.0.4.jar....sy 00000cf0: 732d 7061 636b 6167 652d 6d67 722a 3a20 s.package.mgr... 00000d00: 7072 6f63 6573 7369 6e67 206e 6577 206a processing.new.j 00000d10: 6172 2c20 2743 3a5c 5072 6f67 7261 6d20 ar...C..Program. 00000d20: 4669 6c65 735c 5370 6865 7265 204e 6574 Files.Sphere.Net 00000d30: 776f 726b 735c 4172 656e 6120 506c 6174 works.Arena.Plat 00000d40: 666f 726d 5c70 6c75 6769 6e73 5c61 652e form.plugins.ae. 00000d50: 7370 6865 7265 2e61 7265 6e61 2e6c 6f67 sphere.arena.log 00000d60: 5f32 2e32 2e31 5c6c 6962 5c6c 6f67 346a .2.2.1.lib.log4j 00000d70: 2d31 2e32 2e31 322e 6a61 7227 0d0a 2a73 .1.2.12.jar....s 00000d80: 7973 2d70 6163 6b61 6765 2d6d 6772 2a3a ys.package.mgr.. 00000d90: 2070 726f 6365 7373 696e 6720 6e65 7720 .processing.new. 00000da0: 6a61 722c 2027 433a 5c50 726f 6772 616d jar...C..Program 00000db0: 2046 696c 6573 5c53 7068 6572 6520 4e65 .Files.Sphere.Ne 00000dc0: 7477 6f72 6b73 5c41 7265 6e61 2050 6c61 tworks.Arena.Pla 00000dd0: 7466 6f72 6d5c 706c 7567 696e 735c 6165 tform.plugins.ae 00000de0: 2e73 7068 6572 652e 6172 656e 612e 6e65 .sphere.arena.ne 00000df0: 7477 6f72 6b4d 616e 6167 6572 2e64 6973 tworkManager.dis 00000e00: 636f 7665 7279 5f32 2e33 2e30 5c64 6973 covery.2.3.0.dis 00000e10: 636f 7665 7279 2e6a 6172 270d 0a2a 7379 covery.jar....sy 00000e20: 732d 7061 636b 6167 652d 6d67 722a 3a20 s.package.mgr... 00000e30: 7072 6f63 6573 7369 6e67 206e 6577 206a processing.new.j 00000e40: 6172 2c20 2743 3a5c 5072 6f67 7261 6d20 ar...C..Program. 00000e50: 4669 6c65 735c 5370 6865 7265 204e 6574 Files.Sphere.Net 00000e60: 776f 726b 735c 4172 656e 6120 506c 6174 works.Arena.Plat 00000e70: 666f 726d 5c70 6c75 6769 6e73 5c61 652e form.plugins.ae. 00000e80: 7370 6865 7265 2e61 7265 6e61 2e73 6572 sphere.arena.ser 00000e90: 7665 725f 322e 322e 315c 6172 656e 6153 ver.2.2.1.arenaS 00000ea0: 6572 7665 722e 6a61 7227 0d0a 2a73 7973 erver.jar....sys 00000eb0: 2d70 6163 6b61 6765 2d6d 6772 2a3a 2070 .package.mgr...p 00000ec0: 726f 6365 7373 696e 6720 6e65 7720 6a61 rocessing.new.ja 00000ed0: 722c 2027 433a 5c50 726f 6772 616d 2046 r...C..Program.F 00000ee0: 696c 6573 5c53 7068 6572 6520 4e65 7477 iles.Sphere.Netw 00000ef0: 6f72 6b73 5c41 7265 6e61 2050 6c61 7466 orks.Arena.Platf 00000f00: 6f72 6d5c 706c 7567 696e 735c 6165 2e73 orm.plugins.ae.s 00000f10: 7068 6572 652e 6172 656e 612e 636f 6d6d phere.arena.comm 00000f20: 6f6e 5f32 2e32 2e31 5c63 6f6d 6d6f 6e2e on.2.2.1.common. 00000f30: 6a61 7227 0d0a 2a73 7973 2d70 6163 6b61 jar....sys.packa 00000f40: 6765 2d6d 6772 2a3a 2070 726f 6365 7373 ge.mgr...process 00000f50: 696e 6720 6e65 7720 6a61 722c 2027 433a ing.new.jar...C. 00000f60: 5c50 726f 6772 616d 2046 696c 6573 5c53 .Program.Files.S 00000f70: 7068 6572 6520 4e65 7477 6f72 6b73 5c41 phere.Networks.A 00000f80: 7265 6e61 2050 6c61 7466 6f72 6d5c 706c rena.Platform.pl 00000f90: 7567 696e 735c 6165 2e73 7068 6572 652e ugins.ae.sphere. 00000fa0: 6172 656e 612e 636f 6d6d 6f6e 5f32 2e32 arena.common.2.2 00000fb0: 2e31 5c6c 6962 5c63 6f6d 6d6f 6e73 2d6c .1.lib.commons.l 00000fc0: 616e 672d 322e 332e 6a61 7227 0d0a 2a73 ang.2.3.jar....s 00000fd0: 7973 2d70 6163 6b61 6765 2d6d 6772 2a3a ys.package.mgr.. 00000fe0: 2070 726f 6365 7373 696e 6720 6e65 7720 .processing.new. 00000ff0: 6a61 722c 2027 433a 5c50 726f 6772 616d jar...C..Program at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source) at org.apache.derby.jdbc.Driver30.newEmbedPreparedStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask.run(GooGooStatementCache.java:525) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
          Hide
          Kristian Waagan added a comment -

          Can you explain what you mean with running two DBs in embedded mode?
          What I want to make sure of, is that you don't run two Derby instances (two JVMs) against a single database directory.

          Also, have you tried using the new 10.4.1.3 release?
          If you try, start with a fresh database. Upgrading Derby will not fix the database if it has already been corrupted.

          Show
          Kristian Waagan added a comment - Can you explain what you mean with running two DBs in embedded mode? What I want to make sure of, is that you don't run two Derby instances (two JVMs) against a single database directory. Also, have you tried using the new 10.4.1.3 release? If you try, start with a fresh database. Upgrading Derby will not fix the database if it has already been corrupted.
          Hide
          Shahbaz added a comment -

          Its not the same issue as 3347, as for our use case it happens only when we try to restart our application. We are running two DBs in embedded mode.

          Show
          Shahbaz added a comment - Its not the same issue as 3347, as for our use case it happens only when we try to restart our application. We are running two DBs in embedded mode.
          Hide
          Dyre Tjeldvoll added a comment -

          Downgrading this under the assumption that it is a duplicate of DERBY-3347

          Show
          Dyre Tjeldvoll added a comment - Downgrading this under the assumption that it is a duplicate of DERBY-3347

            People

            • Assignee:
              Unassigned
              Reporter:
              Shahbaz
            • Votes:
              3 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development