[GEODE-3718] The InternalResourceManager fails to shutdown if a redundancy recovery task is scheduled but hasn't fired yet

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.10.0
• Component/s: core
• Labels: needs-review pull-request-available recovery
• Assignee: Mario Ivanac
• Reporter: Barrett Oglesby
• Created: 28/Sep/17 19:51
• Updated: 26/Sep/19 18:05
• Resolved: 15/Jul/19 16:26

Description

This happens with recovery-delay or startup-recovery-delay > 0.

The thread gets stuck here:

"Thread-20" #133 prio=10 os_prio=31 tid=0x00007fa85b886000 nid=0x890b waiting on condition [0x000070001269e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000007bc408900> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1465)
	at org.apache.geode.internal.cache.control.InternalResourceManager.stopExecutor(InternalResourceManager.java:375)
	at org.apache.geode.internal.cache.control.InternalResourceManager.close(InternalResourceManager.java:187)
	at org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2161)
	- locked <0x00000007bc0bc520> (a java.lang.Class for org.apache.geode.internal.cache.GemFireCacheImpl)
	at org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1367)
	- locked <0x00000007bc0bc520> (a java.lang.Class for org.apache.geode.internal.cache.GemFireCacheImpl)
	at org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1017)
	at org.apache.geode.management.internal.beans.MemberMBeanBridge$1.run(MemberMBeanBridge.java:986)
	at java.lang.Thread.run(Thread.java:745)

The InternalResourceManager is waiting for the termination of its scheduledExecutor.

The PRHARedundancyProvider initializes its recoveryExecutor using the InternalResourceManager's scheduledExecutor:

recoveryExecutor = new OneTaskOnlyExecutor(resourceManager.getExecutor(),
  new OneTaskOnlyExecutor.ConflatedTaskListener() {
    public void taskDropped() {
      InternalResourceManager.getResourceObserver().recoveryConflated(region);
    }
  });

The scheduleRedundancyRecovery method schedules a RecoveryRunnable if necessary.

If that task hasn't fired yet, the InternalResourceManager doesn't close, and the JVM stays up.

Issue Links

• links to: GitHub Pull Request #3655
• links to: GitHub Pull Request #3821