Some rawStoreDaemon threads are not stopped

• Key: DERBY-594
• Type: Bug
• Status: Open
• Priority: Minor
• Assignee: Unassigned
• Reporter: Knut Anders Hatlen
• Component/s: Services
• Affects Version/s: 10.1.2.1, 10.2.1.6

Description

In some cases, loading a database will create a rawStoreDaemon thread
which will run forever, even after Derby is shut down with
DriverManager.getConnection("jdbc:derby:;shutdown=true").

I have found two tests in derbyall (there are probably more) where
threads are left running:

  - in store/rollForwardBackup.sql two threads are never terminated

  - in store/encryptDatabase.sql five threads are never terminated

All the threads are named "derby.rawStoreDaemon" and created in
org.apache.derby.impl.store.raw.RawStore.boot().

It seems like this happens in some (but not all) cases where the
loading of a database fails, but I am not able to say exactly what
triggers this bug yet.

Comments

[Knut Anders Hatlen]

I haven't had time to work on this issue for a long time. Just thought
I'd summarize what I have found out till now in case someone is
interested.

TopService.bootModule() has a try/catch like this:

    try {
        BaseMonitor.boot(instance, create, properties);
    } catch (StandardException se) {
        moduleInstances.removeElement(module);
        throw se;
    }

If an exception is thrown in BaseMonitor.boot(), daemon threads might
have been started, but they are not shut down. This can be solved by
putting

    TopService.stop(instance);

in the catch block.

However, this leads to two problems:

  1) LogToFile.stop() gets a NullPointerException in
     deleteObsoleteLogfiles() because currentCheckpoint is null. This
     is easily solved by adding a test for currentCheckpoint == null
     and not calling deleteObsoleteLogfiles() if that's the case.

  2) If one tries to boot a database that is already booted in another
     JVM, BaseMonitor.boot() will fail. However, the call to
     TopService.stop() will delete the boot lock, even though it
     belongs to another process. Therefore, trying once more to boot
     the database will succeed. The solution to this problem is
     probably to set a flag (e.g., ownerOfBootLock) when creating the
     boot lock file, and to check the value of the flag before
     deleting the file.

[Daniel John Debrunner]

Possible blocks DERBY-538, during some tests with my changes for 538 backups are failing due to the JarClassLoader keeping the jar file open of the shutdown database. So something is keeping a reference to database objects, haven't investigated it yet.

[Daniel John Debrunner]

Confirmed that this is not an issue for DERBY-538, the raw store daemon thread stops correctly in the situation I'm seeing.

[Knut Anders Hatlen]

Unassigning the issue since I am not working on it.

[Mike Matrigali]

Triaged July 2, 2009: assigned normal urgency.