Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-8440

Newly restarted node hang in creating region

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.14.0
    • None
    • regions, wan
    • None

    Description

      In one of the internal test we observed test hanging with:

      "vm_9_thr_19_bridge5_host1_31792" #943 daemon prio=5 os_prio=0 cpu=1036.73ms elapsed=1026.10s tid=0x00007fc30c03f000 nid=0x5cbb waiting on condition  [0x00007fc2d1ba2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.8/Native Method)
        - parking to wait for  <0x00000000e2394520> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.8/LockSupport.java:234)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(java.base@11.0.8/AbstractQueuedSynchronizer.java:1079)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(java.base@11.0.8/AbstractQueuedSynchronizer.java:1369)
        at java.util.concurrent.CountDownLatch.await(java.base@11.0.8/CountDownLatch.java:278)
        at org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
        at org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:723)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:794)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:857)
        at org.apache.geode.internal.cache.persistence.MembershipFlushRequest.send(MembershipFlushRequest.java:63)
        at org.apache.geode.internal.cache.PRHARedundancyProvider.recoverPersistentBuckets(PRHARedundancyProvider.java:1598)
        at org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:1110)
        - locked <0x00000000ec1a8218> (a org.apache.geode.internal.cache.PartitionedRegion)
        at org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1220)
        at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3081)
        at org.apache.geode.cache.lucene.internal.LuceneIndexImpl.createRegion(LuceneIndexImpl.java:274)
        at org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createRegion(LuceneIndexForPartitionedRegion.java:186)
        at org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createLuceneListenersAndFileChunkRegions(LuceneIndexForPartitionedRegion.java:123)
        at org.apache.geode.cache.lucene.internal.LuceneIndexImpl.initialize(LuceneIndexImpl.java:150)
        at org.apache.geode.cache.lucene.internal.LuceneServiceImpl.afterDataRegionCreated(LuceneServiceImpl.java:372)
        at org.apache.geode.cache.lucene.internal.LuceneRegionListener.afterCreate(LuceneRegionListener.java:116)
        at org.apache.geode.internal.cache.GemFireCacheImpl.invokeRegionAfter(GemFireCacheImpl.java:3146)
        at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3123)
        at org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:2975)
        at org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:237)
        at org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:675)
        at org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:594)
        at org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:339)
        at org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4180)
        at org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1601)
        at org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1427)
        at org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:191)
        - locked <0x00000000e12f5db8> (a java.lang.Class for org.apache.geode.internal.cache.GemFireCacheImpl)
        - locked <0x00000000e12f5be8> (a java.lang.Class for org.apache.geode.internal.cache.InternalCacheBuilder)
        at org.apache.geode.internal.cache.CacheFactoryStatics.create(CacheFactoryStatics.java:61)
        at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:352)
        at hydra.CacheHelper.createCacheFromXml(CacheHelper.java:215)
        - locked <0x00000000e1302218> (a java.lang.Class for hydra.CacheHelper)
        at util.CacheUtil.createRegion(CacheUtil.java:376)
        at parReg.ParRegTest.HA_reinitializeRegion(ParRegTest.java:806)
        at parReg.ParRegTest.HAController(ParRegTest.java:2050)
        at parReg.ParRegTest.HydraTask_HAController(ParRegTest.java:1077)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.8/Native Method)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.8/NativeMethodAccessorImpl.java:62)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.8/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.8/Method.java:566)
        at hydra.MethExecutor.execute(MethExecutor.java:173)
        at hydra.MethExecutor.execute(MethExecutor.java:141)
        at hydra.TestTask.execute(TestTask.java:197)
        at hydra.RemoteTestModule$1.run(RemoteTestModule.java:213)

*** Stack changed
"vm_9_thr_19_bridge5_host1_31792" #943 daemon prio=5 os_prio=0 cpu=1036.89ms elapsed=1036.12s tid=0x00007fc30c03f000 nid=0x5cbb waiting on condition  [0x00007fc2d1ba2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.8/Native Method)
        - parking to wait for  <0x00000000e2394520> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.8/LockSupport.java:234)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(java.base@11.0.8/AbstractQueuedSynchronizer.java:1079)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(java.base@11.0.8/AbstractQueuedSynchronizer.java:1369)
        at java.util.concurrent.CountDownLatch.await(java.base@11.0.8/CountDownLatch.java:278)
        at org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
        at org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:723)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:794)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:857)
        at org.apache.geode.internal.cache.persistence.MembershipFlushRequest.send(MembershipFlushRequest.java:63)
        at org.apache.geode.internal.cache.PRHARedundancyProvider.recoverPersistentBuckets(PRHARedundancyProvider.java:1598)
        at org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:1110)
        - locked <0x00000000ec1a8218> (a org.apache.geode.internal.cache.PartitionedRegion)
        at org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1220)
        at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3081)
        at org.apache.geode.cache.lucene.internal.LuceneIndexImpl.createRegion(LuceneIndexImpl.java:274)
        at org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createRegion(LuceneIndexForPartitionedRegion.java:186)
        at org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createLuceneListenersAndFileChunkRegions(LuceneIndexForPartitionedRegion.java:123)
        at org.apache.geode.cache.lucene.internal.LuceneIndexImpl.initialize(LuceneIndexImpl.java:150)
        at org.apache.geode.cache.lucene.internal.LuceneServiceImpl.afterDataRegionCreated(LuceneServiceImpl.java:372)
        at org.apache.geode.cache.lucene.internal.LuceneRegionListener.afterCreate(LuceneRegionListener.java:116)
        at org.apache.geode.internal.cache.GemFireCacheImpl.invokeRegionAfter(GemFireCacheImpl.java:3146)
        at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3123)
        at org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:2975)
        at org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:237)
        at org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:675)
        at org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:594)
        at org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:339)
        at org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4180)
        at org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1601)
        at org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1427)
        at org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:191)
        - locked <0x00000000e12f5db8> (a java.lang.Class for org.apache.geode.internal.cache.GemFireCacheImpl)
        - locked <0x00000000e12f5be8> (a java.lang.Class for org.apache.geode.internal.cache.InternalCacheBuilder)
        at org.apache.geode.internal.cache.CacheFactoryStatics.create(CacheFactoryStatics.java:61)
        at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:352)
        at hydra.CacheHelper.createCacheFromXml(CacheHelper.java:215)
        - locked <0x00000000e1302218> (a java.lang.Class for hydra.CacheHelper)
        at util.CacheUtil.createRegion(CacheUtil.java:376)
        at parReg.ParRegTest.HA_reinitializeRegion(ParRegTest.java:806)
        at parReg.ParRegTest.HAController(ParRegTest.java:2050)
        at parReg.ParRegTest.HydraTask_HAController(ParRegTest.java:1077)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.8/Native Method)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.8/NativeMethodAccessorImpl.java:62)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.8/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.8/Method.java:566)
        at hydra.MethExecutor.execute(MethExecutor.java:173)
        at hydra.MethExecutor.execute(MethExecutor.java:141)
        at hydra.TestTask.execute(TestTask.java:197)
        at hydra.RemoteTestModule$1.run(RemoteTestModule.java:213)

*** Stack changed
"vm_9_thr_19_bridge5_host1_31792" #943 daemon prio=5 os_prio=0 cpu=1037.46ms elapsed=1075.21s tid=0x00007fc30c03f000 nid=0x5cbb waiting on condition  [0x00007fc2d1ba2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.8/Native Method)
        - parking to wait for  <0x00000000e2394520> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.8/LockSupport.java:234)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(java.base@11.0.8/AbstractQueuedSynchronizer.java:1079)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(java.base@11.0.8/AbstractQueuedSynchronizer.java:1369)
        at java.util.concurrent.CountDownLatch.await(java.base@11.0.8/CountDownLatch.java:278)
        at org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
        at org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:723)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:794)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:857)
        at org.apache.geode.internal.cache.persistence.MembershipFlushRequest.send(MembershipFlushRequest.java:63)
        at org.apache.geode.internal.cache.PRHARedundancyProvider.recoverPersistentBuckets(PRHARedundancyProvider.java:1598)
        at org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:1110)
        - locked <0x00000000ec1a8218> (a org.apache.geode.internal.cache.PartitionedRegion)
        at org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1220)
        at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3081)
        at org.apache.geode.cache.lucene.internal.LuceneIndexImpl.createRegion(LuceneIndexImpl.java:274)
        at org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createRegion(LuceneIndexForPartitionedRegion.java:186)
        at org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createLuceneListenersAndFileChunkRegions(LuceneIndexForPartitionedRegion.java:123)
        at org.apache.geode.cache.lucene.internal.LuceneIndexImpl.initialize(LuceneIndexImpl.java:150)
        at org.apache.geode.cache.lucene.internal.LuceneServiceImpl.afterDataRegionCreated(LuceneServiceImpl.java:372)
        at org.apache.geode.cache.lucene.internal.LuceneRegionListener.afterCreate(LuceneRegionListener.java:116)
        at org.apache.geode.internal.cache.GemFireCacheImpl.invokeRegionAfter(GemFireCacheImpl.java:3146)
        at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3123)
        at org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:2975)
        at org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:237)
        at org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:675)
        at org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:594)
        at org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:339)
        at org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4180)
        at org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1601)
        at org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1427)
        at org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:191)
        - locked <0x00000000e12f5db8> (a java.lang.Class for org.apache.geode.internal.cache.GemFireCacheImpl)
        - locked <0x00000000e12f5be8> (a java.lang.Class for org.apache.geode.internal.cache.InternalCacheBuilder)
        at org.apache.geode.internal.cache.CacheFactoryStatics.create(CacheFactoryStatics.java:61)
        at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:352)
        at hydra.CacheHelper.createCacheFromXml(CacheHelper.java:215)
        - locked <0x00000000e1302218> (a java.lang.Class for hydra.CacheHelper)
        at util.CacheUtil.createRegion(CacheUtil.java:376)
        at parReg.ParRegTest.HA_reinitializeRegion(ParRegTest.java:806)
        at parReg.ParRegTest.HAController(ParRegTest.java:2050)
        at parReg.ParRegTest.HydraTask_HAController(ParRegTest.java:1077)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.8/Native Method)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.8/NativeMethodAccessorImpl.java:62)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.8/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.8/Method.java:566)
        at hydra.MethExecutor.execute(MethExecutor.java:173)
        at hydra.MethExecutor.execute(MethExecutor.java:141)
        at hydra.TestTask.execute(TestTask.java:197)
        at hydra.RemoteTestModule$1.run(RemoteTestModule.java:213)

Stack for hung thread vm_9_thr_19_bridge5_host1_31792 was found 5 times and changed 4 times.

      Attachments

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            agingade Anilkumar Gingade
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Slack

                Issue deployment