[GEODE-7576] BootstrappingFunction should be executed after cache is fully created

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.12.0
• Component/s: functions
• Labels: GeodeCommons
• Assignee: Ernest Burghardt
• Reporter: Eric Shu
• Created: 13/Dec/19 19:13
• Updated: 30/Mar/20 23:32
• Resolved: 19/Dec/19 18:01

Description

The tomcat client server session module test failed:

[warn 2019/12/12 20:57:59.795 PST <ThreadsMonitor> tid=0x10] Thread <39> (0x27) that was executed at <12 Dec 2019 20:55:00 PST> has been stuck for <178.813 seconds> and number of thread monitor iteration <2>
Thread Name <Function Execution Processor3> state <WAITING>
Waiting on <java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@3f580d54>
Owned By <main> with ID <1>
Executor Group <FunctionExecutionPooledExecutor>
Monitored metric <ResourceManagerStats.numThreadsStuck>
Thread Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lockInterruptibly(ReentrantReadWriteLock.java:772)
org.apache.geode.management.internal.beans.ManagementListener.handleEvent(ManagementListener.java:116)
org.apache.geode.distributed.internal.InternalDistributedSystem.notifyResourceEventListeners(InternalDistributedSystem.java:2065)
org.apache.geode.distributed.internal.InternalDistributedSystem.handleResourceEvent(InternalDistributedSystem.java:606)
org.apache.geode.distributed.internal.locks.DLockService.init(DLockService.java:1915)
org.apache.geode.distributed.internal.locks.DLockService.basicCreate(DLockService.java:1892)
org.apache.geode.distributed.internal.locks.DLockService.create(DLockService.java:2710)
org.apache.geode.internal.cache.GemFireCacheImpl.getPartitionedRegionLockService(GemFireCacheImpl.java:1938)
org.apache.geode.internal.cache.DistributedRegion.<init>(DistributedRegion.java:245)
org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3009)
org.apache.geode.modules.util.CreateRegionFunction.createRegionConfigurationMetadataRegion(CreateRegionFunction.java:273)
org.apache.geode.modules.util.CreateRegionFunction.<init>(CreateRegionFunction.java:63)
org.apache.geode.modules.util.BootstrappingFunction.registerFunctions(BootstrappingFunction.java:124)
org.apache.geode.modules.util.BootstrappingFunction.execute(BootstrappingFunction.java:67)
org.apache.geode.internal.cache.MemberFunctionStreamingMessage.process(MemberFunctionStreamingMessage.java:193)
org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:365)
org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:429)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.geode.distributed.internal.ClusterDistributionManager.runUntilShutdown(ClusterDistributionManager.java:961)
org.apache.geode.distributed.internal.ClusterDistributionManager.doFunctionExecutionThread(ClusterDistributionManager.java:815)
org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$52/1112527632.invoke(Unknown Source)
org.apache.geode.internal.logging.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:121)
org.apache.geode.internal.logging.LoggingThreadFactory$$Lambda$42/973936431.run(Unknown Source)
java.lang.Thread.run(Thread.java:748)

Was able to get a thread dump:
"Function Execution Processor3":
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000005c0732318> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lockInterruptibly(ReentrantReadWriteLock.java:772)
  at org.apache.geode.management.internal.beans.ManagementListener.handleEvent(ManagementListener.java:116)
  at org.apache.geode.distributed.internal.InternalDistributedSystem.notifyResourceEventListeners(InternalDistributedSystem.java:2065)
  at org.apache.geode.distributed.internal.InternalDistributedSystem.handleResourceEvent(InternalDistributedSystem.java:606)
  at org.apache.geode.distributed.internal.locks.DLockService.init(DLockService.java:1915)
  at org.apache.geode.distributed.internal.locks.DLockService.basicCreate(DLockService.java:1892)
  at org.apache.geode.distributed.internal.locks.DLockService.create(DLockService.java:2710)
• locked <0x000000071618b718> (a java.util.HashMap)
• locked <0x000000071618b748> (a java.lang.Object)
  at org.apache.geode.internal.cache.GemFireCacheImpl.getPartitionedRegionLockService(GemFireCacheImpl.java:1938)
• locked <0x00000005c05fd490> (a java.lang.Object)
  at org.apache.geode.internal.cache.DistributedRegion.<init>(DistributedRegion.java:245)
  at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3009)
• locked <0x00000005c05fd1d0> (a java.util.HashMap)
  at org.apache.geode.modules.util.CreateRegionFunction.createRegionConfigurationMetadataRegion(CreateRegionFunction.java:273)
  at org.apache.geode.modules.util.CreateRegionFunction.<init>(CreateRegionFunction.java:63)
  at org.apache.geode.modules.util.BootstrappingFunction.registerFunctions(BootstrappingFunction.java:124)
  at org.apache.geode.modules.util.BootstrappingFunction.execute(BootstrappingFunction.java:67)
  at org.apache.geode.internal.cache.MemberFunctionStreamingMessage.process(MemberFunctionStreamingMessage.java:193)
  at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:365)
  at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:429)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at org.apache.geode.distributed.internal.ClusterDistributionManager.runUntilShutdown(ClusterDistributionManager.java:961)
  at org.apache.geode.distributed.internal.ClusterDistributionManager.doFunctionExecutionThread(ClusterDistributionManager.java:815)
  at org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$53/1202042637.invoke(Unknown Source)
  at org.apache.geode.internal.logging.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:121)
  at org.apache.geode.internal.logging.LoggingThreadFactory$$Lambda$43/973936431.run(Unknown Source)
  at java.lang.Thread.run(Thread.java:748)

"main":
at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:2983)

• waiting to lock <0x00000005c05fd1d0> (a java.util.HashMap)
  at org.apache.geode.internal.cache.InternalCacheForClientAccess.createInternalRegion(InternalCacheForClientAccess.java:240)
  at org.apache.geode.management.internal.LocalManager.startLocalManagement(LocalManager.java:156)
• locked <0x0000000715dfc460> (a org.apache.geode.management.internal.LocalManager)
  at org.apache.geode.management.internal.LocalManager.startManager(LocalManager.java:381)
  at org.apache.geode.management.internal.SystemManagementService.init(SystemManagementService.java:161)
  at org.apache.geode.management.internal.SystemManagementService.newSystemManagementService(SystemManagementService.java:123)
  at org.apache.geode.management.internal.BaseManagementService.getManagementService(BaseManagementService.java:80)
• locked <0x000000071566ad68> (a java.util.HashMap)
  at org.apache.geode.management.ManagementService.getManagementService(ManagementService.java:42)
  at org.apache.geode.management.internal.beans.ManagementAdapter.handleCacheCreation(ManagementAdapter.java:150)
  at org.apache.geode.management.internal.beans.ManagementListener.handleEvent(ManagementListener.java:127)
  at org.apache.geode.distributed.internal.InternalDistributedSystem.notifyResourceEventListeners(InternalDistributedSystem.java:2065)
  at org.apache.geode.distributed.internal.InternalDistributedSystem.handleResourceEvent(InternalDistributedSystem.java:606)
  at org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1239)
  at org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:219)
• locked <0x00000005c01adda8> (a java.lang.Class for org.apache.geode.internal.cache.GemFireCacheImpl)
• locked <0x00000005c004cc08> (a java.lang.Class for org.apache.geode.internal.cache.InternalCacheBuilder)
  at org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:171)
• locked <0x00000005c004cc08> (a java.lang.Class for org.apache.geode.internal.cache.InternalCacheBuilder)
  at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:142)
  at org.apache.geode.distributed.internal.DefaultServerLauncherCacheProvider.createCache(DefaultServerLauncherCacheProvider.java:52)
  at org.apache.geode.distributed.ServerLauncher.createCache(ServerLauncher.java:887)
  at org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:803)
  at org.apache.geode.distributed.ServerLauncher.run(ServerLauncher.java:732)
  at org.apache.geode.distributed.ServerLauncher.main(ServerLauncher.java:251)

Issue Links

• links to: GitHub Pull Request #4477