[DERBY-5632] Logical deadlock happened when freezing/unfreezing the database

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 10.8.2.2
• Fix Version/s: 10.10.1.1
• Component/s: Documentation, Services
• Labels: derby_triage10_10
• Environment: Oracle M3000/Solaris 10
• Urgency: Normal
• Bug behavior facts: Seen in production
• Assignee: Knut Anders Hatlen
• Reporter: Brett Bergquist
• Created: 27/Feb/12 19:28
• Updated: 21/Jan/15 00:23
• Resolved: 14/Mar/13 07:52

Description

Tried to make a quick database backup by freezing the database, performing a ZFS snapshot, and then unfreezing the database. The database was frozen but then a connection to the database could not be established to unfreeze the database.

Looking at the stack trace of the network server, , I see 3 threads that are trying to process a connection request. Each of these is waiting on:

at org.apache.derby.impl.store.access.RAMAccessManager.conglomCacheFind(Unknown Source)

• waiting to lock <0xfffffffd3a7fcc68> (a org.apache.derby.impl.services.cache.ConcurrentCache)

That object is owned by:

• locked <0xfffffffd3a7fcc68> (a org.apache.derby.impl.services.cache.ConcurrentCache)
  at org.apache.derby.impl.store.access.RAMTransaction.findExistingConglomerate(Unknown Source)
  at org.apache.derby.impl.store.access.RAMTransaction.openGroupFetchScan(Unknown Source)
  at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.updateIndexStatsMinion(Unknown Source)
  at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.runExplicitly(Unknown Source)
  at org.apache.derby.impl.sql.execute.AlterTableConstantAction.updateStatistics(Unknown Source)

which itself is waiting for the object:

at java.lang.Object.wait(Native Method)

• waiting on <0xfffffffd3ac1d608> (a org.apache.derby.impl.store.raw.log.LogToFile)
  at java.lang.Object.wait(Object.java:485)
  at org.apache.derby.impl.store.raw.log.LogToFile.flush(Unknown Source)
• locked <0xfffffffd3ac1d608> (a org.apache.derby.impl.store.raw.log.LogToFile)
  at org.apache.derby.impl.store.raw.log.LogToFile.flush(Unknown Source)
  at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.flush(Unknown Source)

So basically what I think is happening is that the database is frozen, the statistics are being updated on another thread which has the "org.apache.derby.impl.services.cache.ConcurrentCache" locked and then waits for the LogToFile lock and the connecting threads are waiting to lock "org.apache.derby.impl.services.cache.ConcurrentCache" to connect and these are where the database is going to be unfrozen. Not a deadlock as far as the JVM is concerned but it will never leave this state either.

Issue Links

• is related to: DERBY-6398 SYSCS_FREEZE_DATABASE locks-up if there are large records that haven't been flushed to the disk
• relates to: DERBY-6110 BrokeredConnection.isClosed() may open a new connection

Sub-Tasks

• Document that freeze/unfreeze must be done from the same connection: Closed