[DERBY-7007] Random IOException: Bad file descriptor on new server platform

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Cannot Reproduce
• Affects Version/s: 10.12.1.1
• Component/s: Miscellaneous

Environment

Linux: SUSE Linux Enterprise Server for SAP Applications 12 SP3 (x86_64)

Kernel: 4.4.126-94.22-default #1 SMP Wed Apr 11 07:45:03 UTC 2018 (9649989) x86_64 x86_64 x86_64 GNU/Linux

Filesystem: /dev/mapper/appsvg-lvapps on /opt/apps type ext3 (rw,relatime,data=ordered)

Java: IBM 7.1-4.15
Tomcat: 7.0.85

• Urgency: Normal
• Bug behavior facts: Crash, Data corruption, Seen in production, Wrong query result
• Flags: Important
• Assignee: Unassigned
• Reporter: Ralf Schubert
• Created: 21/Aug/18 08:58
• Updated: 13/Jan/20 15:54
• Resolved: 13/Jan/20 15:54

Description

Our customer is migrating to a new server platform. We have running several applications on their old server platform right now, which are running well so far. But on the new platform some random Derby errors occur reproducably which we and customer are analysing since several months now. However, the deeper we get the more clueless we are and it looks more and more like a DERBY bug.

We would be pleased if somebody could look into this and give us some idea if this is either a bug in derby or if you have some other ideas what could cause derby to behave like this.

Situation

We have one Application which includes several embedded DERBY databases. After the server is starting, the application behaves normal for a few minutes. But after some minutes, one of the Derby DBs (accessed by JAVA Hibernate using DERBY embedded mode) shows first an error like this on a random derby file (the files vary each time):

Local derby log (/home/tomcat_i36/derby.log):
 
------------  Begin Shutdown Error Stack -------------

ERROR XSDG3: Meta-data for Container(0, 33904) could not be accessed to clean /opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat
        at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
        at org.apache.derby.impl.store.raw.data.RAFContainer.clean(Unknown Source)
        at org.apache.derby.impl.services.cache.ConcurrentCache.cleanAndUnkeepEntry(Unknown Source)
        at org.apache.derby.impl.services.cache.ConcurrentCache.cleanEntry(Unknown Source)
        at org.apache.derby.impl.services.cache.BackgroundCleaner.performWork(Unknown Source)
        at org.apache.derby.impl.services.daemon.BasicDaemon.serviceClient(Unknown Source)
        at org.apache.derby.impl.services.daemon.BasicDaemon.work(Unknown Source)
        at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:809)

Caused by: java.io.IOException: Bad file descriptor
        at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
        at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:65)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
        at sun.nio.ch.IOUtil.read(IOUtil.java:210)
        at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:754)
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:739)
        at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(Unknown Source)
        at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(Unknown Source)
        at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown Source)
        at org.apache.derby.impl.store.raw.data.RAFContainer4.getEmbryonicPage(Unknown Source)
        at org.apache.derby.impl.store.raw.data.RAFContainer.writeRAFHeader(Unknown Source)
        ... 8 more

============= begin nested exception, level (1) ===========

java.io.IOException: Bad file descriptor
        at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
        at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:65)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
        at sun.nio.ch.IOUtil.read(IOUtil.java:210)
        at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:754)
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:739)
        at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(Unknown Source)
        at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(Unknown Source)
        at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown Source)
        at org.apache.derby.impl.store.raw.data.RAFContainer4.getEmbryonicPage(Unknown Source)
        at org.apache.derby.impl.store.raw.data.RAFContainer.writeRAFHeader(Unknown Source)
        at org.apache.derby.impl.store.raw.data.RAFContainer.clean(Unknown Source)
        at org.apache.derby.impl.services.cache.ConcurrentCache.cleanAndUnkeepEntry(Unknown Source)
        at org.apache.derby.impl.services.cache.ConcurrentCache.cleanEntry(Unknown Source)
        at org.apache.derby.impl.services.cache.BackgroundCleaner.performWork(Unknown Source)
        at org.apache.derby.impl.services.daemon.BasicDaemon.serviceClient(Unknown Source)
        at org.apache.derby.impl.services.daemon.BasicDaemon.work(Unknown Source)
        at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:809)

============= end nested exception, level (1) ===========

------------  End Shutdown Error Stack ------------

After this happens, the DB behaves weird, throwing random errors (e.g. telling a column is missing in a table although it is there, or telling the DB is corrupt).

Hint: We do only have READ access on those databases within the application. We do not write any data to it.

It only happens to one single DB, but this is the most complex one in the application. Restarting the server will make it WORK for some minutes again!

We deploy the exact same WAR file to Old and new platform for testing.

Already analysed

We already tried several things and did several analysis steps:

1. Turning off antivirus solution (Trend Micro Deep Security Agent) did not help
2. Exchanging the servers of the new server platform with another set of servers with same setup  does not help
3. Comparing a SHA1 hash of the "corrupt" files with the original files turned out the files are IDENTICAL.
4. Copying the "corrupt" DB to another system, testing it there works as expected without issues.
5. Running an integrity check on the DB shows no problems
6. Checking the file permissions on the problematic servers shows no problems

   # ls -l /opt/apps/tomcat/i36/webapps/XXXX/database/XX/seg0/c8470.dat
   
   -rw-r--r-- 1 tomcat_i36 tomcat 16384 Aug 21 09:32 /opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat
    
   
   # file /opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat
   
   /opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat: data

7. Checking if any linux limits (e.g. open files limit) was reached: nothing found
8. Checking for corrupt file system: Ext3 is used on old and new platform, no hint about corrupt files found
9. Upgrading DERBY from 10.11.1.1 to 10.12.1.1 did not fix the issue.

The server environments

OLD environment (working well)

Linux: SUSE Linux Enterprise Server 11 SP4  (s390x)

Kernel: 3.0.101-91-default #1 SMP Mon Dec 12 13:06:13 UTC 2016 (544b9d1) s390x s390x s390x GNU/Linux

Filesystem: /dev/mapper/appsvg-lvapps on /opt/apps type ext3 (rw,acl,user_xattr)

Java: IBM 7.1-4.1
Tomcat: 7.0.70

NEW environment (not working)

Linux: SUSE Linux Enterprise Server for SAP Applications 12 SP3  (x86_64)

Kernel: 4.4.126-94.22-default #1 SMP Wed Apr 11 07:45:03 UTC 2018 (9649989) x86_64 x86_64 x86_64 GNU/Linux

Filesystem: /dev/mapper/appsvg-lvapps on /opt/apps type ext3 (rw,relatime,data=ordered)

Java: IBM 7.1-4.15
Tomcat: 7.0.85

Our customer has to migrate the server platforms very soon so we would be very glad if someone could assist us in checking and resolving this.

Thanks!

Issue Links

• relates to: DERBY-4139 error on sql when upgrading database from 10.2.2 to 10.4.1.3