Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-8206

CI Failure: ReconnectWithClusterConfigurationDUnitTest.testReconnectAfterMeltdown hang

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.14.0
    • ci, membership
    • None

    Description

      This test hangs in: https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/DistributedTestOpenJDK8/builds/221#A

      "RMI TCP Connection(1)-172.17.0.23" #32 daemon prio=5 os_prio=0 tid=0x00007fea58001800 nid=0x27d waiting on condition [0x00007feb23bf7000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000000e0e6cb10> (a java.util.concurrent.CountDownLatch$Sync)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
              at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
              at org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
              at org.apache.geode.internal.cache.LocalRegion.waitOnInitialization(LocalRegion.java:4693)
              at org.apache.geode.internal.cache.LocalRegion.waitOnInitialization(LocalRegion.java:4671)
              at org.apache.geode.internal.cache.LocalRegion.getSubregion(LocalRegion.java:4558)
              at org.apache.geode.internal.cache.GemFireCacheImpl.getRegion(GemFireCacheImpl.java:3322)
              at org.apache.geode.internal.cache.GemFireCacheImpl.getRegion(GemFireCacheImpl.java:3153)
              at org.apache.geode.distributed.internal.InternalConfigurationPersistenceService.getConfigurationRegion(InternalConfigurationPersistenceService.java:792)
              at org.apache.geode.distributed.internal.InternalConfigurationPersistenceService.destroySharedConfiguration(InternalConfigurationPersistenceService.java:639)
              at org.apache.geode.cache30.ReconnectWithClusterConfigurationDUnitTest.lambda$teardown$bb17a952$1(ReconnectWithClusterConfigurationDUnitTest.java:112)
              at org.apache.geode.cache30.ReconnectWithClusterConfigurationDUnitTest$$Lambda$339/1176989958.run(Unknown Source)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.apache.geode.test.dunit.internal.MethodInvoker.executeObject(MethodInvoker.java:123)
              at org.apache.geode.test.dunit.internal.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:78)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
              at sun.rmi.transport.Transport$1.run(Transport.java:200)
              at sun.rmi.transport.Transport$1.run(Transport.java:197)
              at java.security.AccessController.doPrivileged(Native Method)
              at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
              at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
              at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
              at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
              at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$15/2049270388.run(Unknown Source)
              at java.security.AccessController.doPrivileged(Native Method)
              at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
         Locked ownable synchronizers:
              - <0x00000000e0a75928> (a java.util.concurrent.ThreadPoolExecutor$Worker)
      
      

      It is waiting for initializationLatchAfterGetInitialImage to countdown. Not sure why gii was not able to finish.

      There is another reconnect thread is waiting in another process. I do not have enough context to tell if it is the gii provider or not.

      "ReconnectThread" #77 prio=5 os_prio=0 tid=0x00007f42281f7000 nid=0x353 in Object.wait() [0x00007f43082db000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2568)
              at org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2423)
              - locked <0x00000000e0867ca8> (a java.lang.Object)
              - locked <0x00000000e08b1778> (a java.lang.Class for org.apache.geode.internal.cache.GemFireCacheImpl)
              - locked <0x00000000e0867ce8> (a java.lang.Class for org.apache.geode.internal.cache.InternalCacheBuilder)
              at org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1275)
              at org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:2315)
              at org.apache.geode.distributed.internal.membership.gms.GMSMembership.uncleanShutdown(GMSMembership.java:1287)
              at org.apache.geode.distributed.internal.membership.gms.GMSMembership$ManagerImpl.lambda$forceDisconnect$0(GMSMembership.java:2030)
              at org.apache.geode.distributed.internal.membership.gms.GMSMembership$ManagerImpl$$Lambda$261/2070151959.run(Unknown Source)
              at java.lang.Thread.run(Thread.java:748)
      
         Locked ownable synchronizers:
              - None
      

      Attachments

        Issue Links

          Activity

            People

              bschuchardt Bruce J Schuchardt
              eshu Eric Shu
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: