Affects Version/s: 10.12.1.1
Fix Version/s: None
Environment:Linux: SUSE Linux Enterprise Server for SAP Applications 12 SP3 (x86_64)
Kernel: 4.4.126-94.22-default #1 SMP Wed Apr 11 07:45:03 UTC 2018 (9649989) x86_64 x86_64 x86_64 GNU/Linux
Filesystem: /dev/mapper/appsvg-lvapps on /opt/apps type ext3 (rw,relatime,data=ordered)
Java: IBM 7.1-4.15
Tomcat: 7.0.85Linux: SUSE Linux Enterprise Server for SAP Applications 12 SP3 (x86_64) Kernel: 4.4.126-94.22-default #1 SMP Wed Apr 11 07:45:03 UTC 2018 (9649989) x86_64 x86_64 x86_64 GNU/Linux Filesystem: /dev/mapper/appsvg-lvapps on /opt/apps type ext3 (rw,relatime,data=ordered) Java: IBM 7.1-4.15 Tomcat: 7.0.85
Bug behavior facts:Crash, Data corruption, Seen in production, Wrong query result
Our customer is migrating to a new server platform. We have running several applications on their old server platform right now, which are running well so far. But on the new platform some random Derby errors occur reproducably which we and customer are analysing since several months now. However, the deeper we get the more clueless we are and it looks more and more like a DERBY bug.
We would be pleased if somebody could look into this and give us some idea if this is either a bug in derby or if you have some other ideas what could cause derby to behave like this.
We have one Application which includes several embedded DERBY databases. After the server is starting, the application behaves normal for a few minutes. But after some minutes, one of the Derby DBs (accessed by JAVA Hibernate using DERBY embedded mode) shows first an error like this on a random derby file (the files vary each time):
After this happens, the DB behaves weird, throwing random errors (e.g. telling a column is missing in a table although it is there, or telling the DB is corrupt).
Hint: We do only have READ access on those databases within the application. We do not write any data to it.
It only happens to one single DB, but this is the most complex one in the application. Restarting the server will make it WORK for some minutes again!
We deploy the exact same WAR file to Old and new platform for testing.
We already tried several things and did several analysis steps:
- Turning off antivirus solution (Trend Micro Deep Security Agent) did not help
- Exchanging the servers of the new server platform with another set of servers with same setup does not help
- Comparing a SHA1 hash of the "corrupt" files with the original files turned out the files are IDENTICAL.
- Copying the "corrupt" DB to another system, testing it there works as expected without issues.
- Running an integrity check on the DB shows no problems
- Checking the file permissions on the problematic servers shows no problems
- Checking if any linux limits (e.g. open files limit) was reached: nothing found
- Checking for corrupt file system: Ext3 is used on old and new platform, no hint about corrupt files found
- Upgrading DERBY from 10.11.1.1 to 10.12.1.1 did not fix the issue.
Our customer has to migrate the server platforms very soon so we would be very glad if someone could assist us in checking and resolving this.