Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-6659

CI failure: DistributedSystemMXBeanWithAlertsDistributedTest > managerReceivesAlertsFromAllMembersAtAlertLevelAndAbove cannot send "cache closed" message

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.13.0
    • Component/s: core, logging, membership
    • Labels:
      None

      Description

      When the cache is closed during test teardown, it can't get a connection to send the "cache closing" message. It then tries to send an alert about that failure, and it can't get a connection to send that either. So it throws IOException and bails out.

      CI failure: https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/DistributedTestOpenJDK11/builds/624

      Test results: http://files.apachegeode-ci.info/builds/apache-develop-main/1.10.0-SNAPSHOT.0183/test-results/distributedTest/1555361324/

      Stack trace:

      java.lang.AssertionError: Suspicious strings were written to the log during this run.
      Fix the strings or use IgnoredException.addIgnoredException to ignore.
      -----------------------------------------------------------------------
      Found suspect string in log4j at line 2457
      
      [fatal 2019/04/15 19:48:11.076 UTC <RMI TCP Connection(1)-172.17.0.3> tid=34] Failed to send message <Alert "Attempting TCP/IP reconnect to  172.17.0.3(managerVM:149)<v28>:41001" level WARNING> to member <172.17.0.3(managerVM:149)<v28>:41001> view, View[172.17.0.3(58:locator)<ec><v0>:41000|31] members: [172.17.0.3(58:locator)<ec><v0>:41000, 172.17.0.3(managerVM:149)<v28>:41001{lead}, 172.17.0.3(memberVM-1:153)<v29>:41002, 172.17.0.3(memberVM-2:158)<v30>:41003, 172.17.0.3(memberVM-3:162)<v31>:41004]
      java.io.IOException: Cannot form connection to alert listener 172.17.0.3(managerVM:149)<v28>:41001
      	at org.apache.geode.internal.tcp.Connection.createSender(Connection.java:952)
      	at org.apache.geode.internal.tcp.ConnectionTable.handleNewPendingConnection(ConnectionTable.java:293)
      	at org.apache.geode.internal.tcp.ConnectionTable.getSharedConnection(ConnectionTable.java:404)
      	at org.apache.geode.internal.tcp.ConnectionTable.get(ConnectionTable.java:589)
      	at org.apache.geode.internal.tcp.TCPConduit.getConnection(TCPConduit.java:824)
      	at org.apache.geode.distributed.internal.direct.DirectChannel.getConnections(DirectChannel.java:536)
      	at org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:326)
      	at org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:241)
      	at org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:595)
      	at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1710)
      	at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1891)
      	at org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2852)
      	at org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:2779)
      	at org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2816)
      	at org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1526)
      	at org.apache.geode.internal.alerting.AlertMessaging.sendAlert(AlertMessaging.java:75)
      	at org.apache.geode.internal.logging.log4j.AlertAppender.sendAlertMessage(AlertAppender.java:187)
      	at org.apache.geode.internal.logging.log4j.AlertAppender.doAppend(AlertAppender.java:162)
      	at org.apache.geode.internal.logging.log4j.AlertAppender.lambda$append$0(AlertAppender.java:158)
      	at org.apache.geode.internal.alerting.AlertingAction.execute(AlertingAction.java:29)
      	at org.apache.geode.internal.logging.log4j.AlertAppender.append(AlertAppender.java:158)
      	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
      	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
      	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
      	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
      	at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:464)
      	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:448)
      	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:431)
      	at org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:455)
      	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:450)
      	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:431)
      	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:406)
      	at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
      	at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
      	at org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(ExtendedLoggerWrapper.java:217)
      	at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2170)
      	at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2125)
      	at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2108)
      	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2019)
      	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1890)
      	at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2747)
      	at org.apache.geode.internal.tcp.TCPConduit.getConnection(TCPConduit.java:792)
      	at org.apache.geode.distributed.internal.direct.DirectChannel.getConnections(DirectChannel.java:536)
      	at org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:326)
      	at org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:597)
      	at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1710)
      	at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1891)
      	at org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2852)
      	at org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:2779)
      	at org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2816)
      	at org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1526)
      	at org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2282)
      	at org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:1907)
      	at org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:1897)
      	at org.apache.geode.management.DistributedSystemMXBeanWithAlertsDistributedTest.lambda$tearDown$bb17a952$1(DistributedSystemMXBeanWithAlertsDistributedTest.java:153)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at org.apache.geode.test.dunit.internal.MethodInvoker.executeObject(MethodInvoker.java:123)
      	at org.apache.geode.test.dunit.internal.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:69)
      	at jdk.internal.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:359)
      	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200)
      	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197)
      	at java.base/java.security.AccessController.doPrivileged(Native Method)
      	at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196)
      	at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562)
      	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796)
      	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677)
      	at java.base/java.security.AccessController.doPrivileged(Native Method)
      	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:834)
      
      
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.apache.geode.test.dunit.internal.DUnitLauncher.closeAndCheckForSuspects(DUnitLauncher.java:380)
      	at org.apache.geode.test.dunit.rules.DistributedRule$TearDown.doTearDown(DistributedRule.java:215)
      	at org.apache.geode.test.dunit.rules.DistributedRule.after(DistributedRule.java:147)
      	at org.apache.geode.test.dunit.rules.AbstractDistributedRule.afterDistributedTest(AbstractDistributedRule.java:81)
      	at org.apache.geode.test.dunit.rules.AbstractDistributedRule$1.evaluate(AbstractDistributedRule.java:61)
      	at org.apache.geode.test.dunit.rules.AbstractDistributedRule$1.evaluate(AbstractDistributedRule.java:59)
      	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
      	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
      	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
      	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
      	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
      	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
      	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:566)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
      	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
      	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
      	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
      	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
      	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:566)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
      	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
      	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
      	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
      	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
      	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
      	at java.lang.Thread.run(Thread.java:834)
      

        Attachments

          Activity

            People

            • Assignee:
              klund Kirk Lund
              Reporter:
              demery Dale Emery
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: