Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-2542

LocatorDUnitTest and LocatorUDPSecurityDUnitTest fail frequently in Geode Nightly Build

Details

    Description

      testMultipleLocators:

      org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedRunnable.run in VM 0 running on Host asf919.gq1.ygridcore.net with 5 VMs
      	at org.apache.geode.test.dunit.VM.invoke(VM.java:377)
      	at org.apache.geode.test.dunit.VM.invoke(VM.java:347)
      	at org.apache.geode.test.dunit.VM.invoke(VM.java:280)
      	at org.apache.geode.distributed.LocatorDUnitTest.testMultipleLocators(LocatorDUnitTest.java:1567)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
      	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
      	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
      	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
      	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
      	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
      	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
      	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
      	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
      	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377)
      	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
      	at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.geode.GemFireConfigException: Unable to join the distributed system.  Operation either timed out, was stopped or Locator does not exist.
      	at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:662)
      	at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:748)
      	at org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:181)
      	at org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:102)
      	at org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:89)
      	at org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1112)
      	at org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1160)
      	at org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:531)
      	at org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:687)
      	at org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)
      	at org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:202)
      	at org.apache.geode.distributed.internal.InternalLocator.startDistributedSystem(InternalLocator.java:739)
      	at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:357)
      	at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:315)
      	at org.apache.geode.distributed.Locator.startLocator(Locator.java:253)
      	at org.apache.geode.distributed.Locator.startLocatorAndDS(Locator.java:140)
      	at org.apache.geode.distributed.LocatorDUnitTest.startLocatorInternal(LocatorDUnitTest.java:624)
      	at org.apache.geode.distributed.LocatorDUnitTest.startLocator(LocatorDUnitTest.java:615)
      	at org.apache.geode.distributed.LocatorDUnitTest.lambda$testMultipleLocators$4f3bdd8a$1(LocatorDUnitTest.java:1567)
      	at org.apache.geode.test.dunit.NamedRunnable.run(NamedRunnable.java:31)
      	at sun.reflect.GeneratedMethodAccessor159.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at hydra.MethExecutor.executeObject(MethExecutor.java:245)
      	at org.apache.geode.test.dunit.standalone.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:73)
      	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
      	at sun.rmi.transport.Transport$1.run(Transport.java:200)
      	at sun.rmi.transport.Transport$1.run(Transport.java:197)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
      	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
      	... 3 more
      

      testLocatorBecomesCoordinator

      org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedRunnable.run in VM 0 running on Host asf902.gq1.ygridcore.net with 5 VMs
      	at org.apache.geode.test.dunit.VM.invoke(VM.java:377)
      	at org.apache.geode.test.dunit.VM.invoke(VM.java:347)
      	at org.apache.geode.test.dunit.VM.invoke(VM.java:280)
      	at org.apache.geode.distributed.LocatorDUnitTest.testLocatorBecomesCoordinator(LocatorDUnitTest.java:1432)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
      	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
      	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
      	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
      	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
      	at sun.reflect.GeneratedMethodAccessor202.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
      	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
      	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
      	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
      	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
      	at sun.reflect.GeneratedMethodAccessor201.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
      	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377)
      	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
      	at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.geode.GemFireConfigException: Unable to join the distributed system.  Operation either timed out, was stopped or Locator does not exist.
      	at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:659)
      	at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:745)
      	at org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:181)
      	at org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:102)
      	at org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:89)
      	at org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1112)
      	at org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1160)
      	at org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:531)
      	at org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:687)
      	at org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)
      	at org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:202)
      	at org.apache.geode.distributed.internal.InternalLocator.startDistributedSystem(InternalLocator.java:739)
      	at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:357)
      	at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:315)
      	at org.apache.geode.distributed.Locator.startLocator(Locator.java:253)
      	at org.apache.geode.distributed.Locator.startLocatorAndDS(Locator.java:140)
      	at org.apache.geode.distributed.LocatorDUnitTest.getStartSBLocatorRunnable(LocatorDUnitTest.java:2079)
      	at org.apache.geode.distributed.LocatorDUnitTest.lambda$testLocatorBecomesCoordinator$2c6907a2$1(LocatorDUnitTest.java:1432)
      	at org.apache.geode.test.dunit.NamedRunnable.run(NamedRunnable.java:31)
      	at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at hydra.MethExecutor.executeObject(MethExecutor.java:245)
      	at org.apache.geode.test.dunit.standalone.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:73)
      	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
      	at sun.rmi.transport.Transport$1.run(Transport.java:200)
      	at sun.rmi.transport.Transport$1.run(Transport.java:197)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
      	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
      	... 3 more
      

      testNonSSLLocatorDiesWhenConnectingToSSLLocator:

      org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedCallable.call in VM 1 running on Host asf902.gq1.ygridcore.net with 5 VMs
      	at org.apache.geode.test.dunit.VM.invoke(VM.java:377)
      	at org.apache.geode.test.dunit.VM.invoke(VM.java:347)
      	at org.apache.geode.test.dunit.VM.invoke(VM.java:304)
      	at org.apache.geode.distributed.LocatorDUnitTest.testNonSSLLocatorDiesWhenConnectingToSSLLocator(LocatorDUnitTest.java:461)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
      	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
      	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
      	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
      	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
      	at sun.reflect.GeneratedMethodAccessor202.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
      	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
      	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
      	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
      	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
      	at sun.reflect.GeneratedMethodAccessor201.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
      	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
      	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377)
      	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
      	at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.AssertionError: expected:<1> but was:<2>
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.failNotEquals(Assert.java:834)
      	at org.junit.Assert.assertEquals(Assert.java:645)
      	at org.junit.Assert.assertEquals(Assert.java:631)
      	at org.apache.geode.distributed.LocatorDUnitTest.expectSystemToContainThisManyMembers(LocatorDUnitTest.java:610)
      	at org.apache.geode.distributed.LocatorDUnitTest.lambda$testNonSSLLocatorDiesWhenConnectingToSSLLocator$515fd116$1(LocatorDUnitTest.java:462)
      	at org.apache.geode.test.dunit.NamedCallable.call(NamedCallable.java:31)
      	at sun.reflect.GeneratedMethodAccessor318.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at hydra.MethExecutor.executeObject(MethExecutor.java:245)
      	at org.apache.geode.test.dunit.standalone.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:73)
      	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
      	at sun.rmi.transport.Transport$1.run(Transport.java:200)
      	at sun.rmi.transport.Transport$1.run(Transport.java:197)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
      	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
      	... 3 more
      

      Attachments

        Issue Links

          Activity

            Commit 8e22ea2137cb58211e0440defe8cbd3e7e11554c in geode's branch refs/heads/develop from apache@the9muses.net
            [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=8e22ea2 ]

            GEODE-2542: add FlakyTest category to LocatorDUnit and LocatorUDPSecurityDUnitTest

            jira-bot ASF subversion and git services added a comment - Commit 8e22ea2137cb58211e0440defe8cbd3e7e11554c in geode's branch refs/heads/develop from apache@the9muses.net [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=8e22ea2 ] GEODE-2542 : add FlakyTest category to LocatorDUnit and LocatorUDPSecurityDUnitTest

            Commit 866dc5ca1583c5fab49ec96c48d261c0367427f3 in geode's branch refs/heads/feature/GEODE-1793 from bschuchardt
            [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=866dc5c ]

            GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL

            This was a product issue. When the locator using plain-text sockets is
            contacted by a TcpClient using SSL the locator often just closes the socket.
            On some platforms this causes a SSLHandshakeException but on others it
            just causes a "SocketException: connection reset". Writing some text to
            the socket forces the TcpClient to get a SSLException (which is the superclass
            of SSLHandshakeException).

            The test class is still marked as Flaky due to GEODE-2542.

            jira-bot ASF subversion and git services added a comment - Commit 866dc5ca1583c5fab49ec96c48d261c0367427f3 in geode's branch refs/heads/feature/ GEODE-1793 from bschuchardt [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=866dc5c ] GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL This was a product issue. When the locator using plain-text sockets is contacted by a TcpClient using SSL the locator often just closes the socket. On some platforms this causes a SSLHandshakeException but on others it just causes a "SocketException: connection reset". Writing some text to the socket forces the TcpClient to get a SSLException (which is the superclass of SSLHandshakeException). The test class is still marked as Flaky due to GEODE-2542 .

            Commit 064362e90e150ad4ef5b269fab78f0cf2d6e5f4f in geode's branch refs/heads/develop from bschuchardt
            [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=064362e ]

            GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL

            This was a product issue. When the locator using plain-text sockets is
            contacted by a TcpClient using SSL the locator often just closes the socket.
            On some platforms this causes a SSLHandshakeException but on others it
            just causes a "SocketException: connection reset". Writing some text to
            the socket forces the TcpClient to get a SSLException (which is the superclass
            of SSLHandshakeException).

            The test class is still marked as Flaky due to GEODE-2542.

            jira-bot ASF subversion and git services added a comment - Commit 064362e90e150ad4ef5b269fab78f0cf2d6e5f4f in geode's branch refs/heads/develop from bschuchardt [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=064362e ] GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL This was a product issue. When the locator using plain-text sockets is contacted by a TcpClient using SSL the locator often just closes the socket. On some platforms this causes a SSLHandshakeException but on others it just causes a "SocketException: connection reset". Writing some text to the socket forces the TcpClient to get a SSLException (which is the superclass of SSLHandshakeException). The test class is still marked as Flaky due to GEODE-2542 .

            commit 0bdfd0e35d8c02f23082f23c2c3552678ccd7421
            Author: Bruce Schuchardt <bschuchardt@pivotal.io>
            Date: Thu Mar 16 10:15:06 2017 -0700

            GEODE-2542 LocatorDUnitTest and LocatorUDPSecurityDUnitTest flakyness

            The fix for GEODE-1793 removed the short join-timeouts from this test class
            that caused periodic failure to join the distributed system.

            bschuchardt Bruce J Schuchardt added a comment - commit 0bdfd0e35d8c02f23082f23c2c3552678ccd7421 Author: Bruce Schuchardt <bschuchardt@pivotal.io> Date: Thu Mar 16 10:15:06 2017 -0700 GEODE-2542 LocatorDUnitTest and LocatorUDPSecurityDUnitTest flakyness The fix for GEODE-1793 removed the short join-timeouts from this test class that caused periodic failure to join the distributed system.

            Commit 3f12fd460b1c34543d56ef4f6844f2b9f56918d5 in geode's branch refs/heads/develop from bschuchardt
            [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=3f12fd4 ]

            GEODE-2542 LocatorDUnitTest and LocatorUDPSecurityDUnitTest flaky

            Removed yet another Flaky annotation from this test class. Flakiness
            was fixed months ago but the Flaky annotations and categorization
            weren't removed from the test.

            jira-bot ASF subversion and git services added a comment - Commit 3f12fd460b1c34543d56ef4f6844f2b9f56918d5 in geode's branch refs/heads/develop from bschuchardt [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=3f12fd4 ] GEODE-2542 LocatorDUnitTest and LocatorUDPSecurityDUnitTest flaky Removed yet another Flaky annotation from this test class. Flakiness was fixed months ago but the Flaky annotations and categorization weren't removed from the test.

            Reopening as the test is still marked Flaky and is still failing even though it's run as part of the flakyTest suite and so has no interference from other unit tests.

            org.apache.geode.distributed.LocatorUDPSecurityDUnitTest > testStartTwoLocators FAILED
                org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedCallable.call in VM 2 running on Host 1fcec927-d828-4bc5-5f68-8f441aa87f30 with 5 VMs
            
                    Caused by:
                    java.lang.AssertionError: expected:<2> but was:<1>
            
                java.lang.AssertionError: Suspicious strings were written to the log during this run.
                Fix the strings or use IgnoredException.addIgnoredException to ignore.
                -----------------------------------------------------------------------
                Found suspect string in log4j at line 2674
            
                [error 2017/06/14 01:38:00.381 UTC <unicast receiver,1fcec927-d828-4bc5-5f68-8f441aa87f30-42434> tid=0x317] Exception deserializing message payload: [dst: 1fcec927-d828-4bc5-5f68-8f441aa87f30<v1>:32771, src: 1fcec927-d828-4bc5-5f68-8f441aa87f30<v0>:32770 (2 headers), size=1452 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER]
                java.lang.Exception: Message id is -150
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1109)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1010)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1278)
                	at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
                	at org.jgroups.JChannel.up(JChannel.java:741)
                	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
                	at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
                	at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
                	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
                	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
                	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
                	at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
                	at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
                	at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
                	at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729)
                	at org.jgroups.protocols.TP.receive(TP.java:1654)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160)
                	at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
                	at java.lang.Thread.run(Thread.java:748)
                Caused by: java.lang.NullPointerException
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterSecretKey(GMSEncrypt.java:103)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterEncryptor(GMSEncrypt.java:290)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.decryptData(GMSEncrypt.java:153)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1088)
                	... 20 more
            
            bschuchardt Bruce J Schuchardt added a comment - Reopening as the test is still marked Flaky and is still failing even though it's run as part of the flakyTest suite and so has no interference from other unit tests. org.apache.geode.distributed.LocatorUDPSecurityDUnitTest > testStartTwoLocators FAILED org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedCallable.call in VM 2 running on Host 1fcec927-d828-4bc5-5f68-8f441aa87f30 with 5 VMs Caused by: java.lang.AssertionError: expected:<2> but was:<1> java.lang.AssertionError: Suspicious strings were written to the log during this run. Fix the strings or use IgnoredException.addIgnoredException to ignore. ----------------------------------------------------------------------- Found suspect string in log4j at line 2674 [error 2017/06/14 01:38:00.381 UTC <unicast receiver,1fcec927-d828-4bc5-5f68-8f441aa87f30-42434> tid=0x317] Exception deserializing message payload: [dst: 1fcec927-d828-4bc5-5f68-8f441aa87f30<v1>:32771, src: 1fcec927-d828-4bc5-5f68-8f441aa87f30<v0>:32770 (2 headers), size=1452 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER] java.lang.Exception: Message id is -150 at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1109) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1010) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1278) at org.jgroups.JChannel.invokeCallback(JChannel.java:816) at org.jgroups.JChannel.up(JChannel.java:741) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030) at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:390) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426) at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74) at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72) at org.jgroups.protocols.TP.passMessageUp(TP.java:1601) at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817) at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10) at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729) at org.jgroups.protocols.TP.receive(TP.java:1654) at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160) at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterSecretKey(GMSEncrypt.java:103) at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterEncryptor(GMSEncrypt.java:290) at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.decryptData(GMSEncrypt.java:153) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1088) ... 20 more

            I am disabling testStartTwoLocators along with a couple of new tests that I'm adding to the superclass in an upcoming checkin.

            bschuchardt Bruce J Schuchardt added a comment - I am disabling testStartTwoLocators along with a couple of new tests that I'm adding to the superclass in an upcoming checkin.

            A different failure is seen in testCollocatedLocatorWithSecurity:

            java.lang.AssertionError: Suspicious strings were written to the log during this run.
            Fix the strings or use IgnoredException.addIgnoredException to ignore.
            -----------------------------------------------------------------------
            Found suspect string in log4j at line 1042
            
            [fatal 2017/06/14 10:48:52.143 PDT <locator request thread[1]> tid=0xbe] Exception in processing request from 10.118.33.201
            java.lang.RuntimeException: Not found public key for member 10.118.33.201(63107)<v1>:32771
            	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getPublicKey(GMSEncrypt.java:177)
            	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.getPublicKey(JGroupsMessenger.java:1367)
            	at org.apache.geode.distributed.internal.membership.gms.locator.GMSLocator.processRequest(GMSLocator.java:283)
            	at org.apache.geode.distributed.internal.InternalLocator$PrimaryHandler.processRequest(InternalLocator.java:1256)
            	at org.apache.geode.distributed.internal.tcpserver.TcpServer.lambda$processRequest$0(TcpServer.java:401)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            	at java.lang.Thread.run(Thread.java:748)
            Caused by: java.lang.NullPointerException
            	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getPeerEncryptor(GMSEncrypt.java:258)
            	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getPublicKey(GMSEncrypt.java:175)
            	... 7 more
            
            bschuchardt Bruce J Schuchardt added a comment - A different failure is seen in testCollocatedLocatorWithSecurity: java.lang.AssertionError: Suspicious strings were written to the log during this run. Fix the strings or use IgnoredException.addIgnoredException to ignore. ----------------------------------------------------------------------- Found suspect string in log4j at line 1042 [fatal 2017/06/14 10:48:52.143 PDT <locator request thread[1]> tid=0xbe] Exception in processing request from 10.118.33.201 java.lang.RuntimeException: Not found public key for member 10.118.33.201(63107)<v1>:32771 at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getPublicKey(GMSEncrypt.java:177) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.getPublicKey(JGroupsMessenger.java:1367) at org.apache.geode.distributed.internal.membership.gms.locator.GMSLocator.processRequest(GMSLocator.java:283) at org.apache.geode.distributed.internal.InternalLocator$PrimaryHandler.processRequest(InternalLocator.java:1256) at org.apache.geode.distributed.internal.tcpserver.TcpServer.lambda$processRequest$0(TcpServer.java:401) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getPeerEncryptor(GMSEncrypt.java:258) at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getPublicKey(GMSEncrypt.java:175) ... 7 more
            klund Kirk Lund added a comment -

            Similar failure but different test method:

            org.apache.geode.distributed.LocatorUDPSecurityDUnitTest > testStartTwoLocators FAILED
                org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedCallable.call in VM 2 running on Host c52f651b-907c-405b-6209-c0d8e0c3c93d with 5 VMs
            
                    Caused by:
                    java.lang.AssertionError: expected:<2> but was:<1>
            
                java.lang.AssertionError: Suspicious strings were written to the log during this run.
                Fix the strings or use IgnoredException.addIgnoredException to ignore.
                -----------------------------------------------------------------------
                Found suspect string in log4j at line 1784
            
                [error 2017/06/16 22:55:51.582 UTC <unicast receiver,c52f651b-907c-405b-6209-c0d8e0c3c93d-12820> tid=0x261] Exception deserializing message payload: [dst: c52f651b-907c-405b-6209-c0d8e0c3c93d<v1>:32771, src: c52f651b-907c-405b-6209-c0d8e0c3c93d<v0>:32770 (2 headers), size=1452 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER]
                java.lang.Exception: Message id is -150
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1109)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1010)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1278)
                	at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
                	at org.jgroups.JChannel.up(JChannel.java:741)
                	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
                	at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
                	at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
                	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
                	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
                	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
                	at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
                	at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
                	at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
                	at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729)
                	at org.jgroups.protocols.TP.receive(TP.java:1654)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160)
                	at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
                	at java.lang.Thread.run(Thread.java:748)
                Caused by: java.lang.NullPointerException
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterSecretKey(GMSEncrypt.java:103)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterEncryptor(GMSEncrypt.java:290)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.decryptData(GMSEncrypt.java:153)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1088)
                	... 20 more
            
            klund Kirk Lund added a comment - Similar failure but different test method: org.apache.geode.distributed.LocatorUDPSecurityDUnitTest > testStartTwoLocators FAILED org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedCallable.call in VM 2 running on Host c52f651b-907c-405b-6209-c0d8e0c3c93d with 5 VMs Caused by: java.lang.AssertionError: expected:<2> but was:<1> java.lang.AssertionError: Suspicious strings were written to the log during this run. Fix the strings or use IgnoredException.addIgnoredException to ignore. ----------------------------------------------------------------------- Found suspect string in log4j at line 1784 [error 2017/06/16 22:55:51.582 UTC <unicast receiver,c52f651b-907c-405b-6209-c0d8e0c3c93d-12820> tid=0x261] Exception deserializing message payload: [dst: c52f651b-907c-405b-6209-c0d8e0c3c93d<v1>:32771, src: c52f651b-907c-405b-6209-c0d8e0c3c93d<v0>:32770 (2 headers), size=1452 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER] java.lang.Exception: Message id is -150 at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1109) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1010) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1278) at org.jgroups.JChannel.invokeCallback(JChannel.java:816) at org.jgroups.JChannel.up(JChannel.java:741) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030) at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:390) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426) at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74) at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72) at org.jgroups.protocols.TP.passMessageUp(TP.java:1601) at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817) at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10) at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729) at org.jgroups.protocols.TP.receive(TP.java:1654) at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160) at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterSecretKey(GMSEncrypt.java:103) at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterEncryptor(GMSEncrypt.java:290) at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.decryptData(GMSEncrypt.java:153) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1088) ... 20 more

            Commit e9f856357565e72e8fe7dff7235404a74fdb5af9 in geode's branch refs/heads/develop from WireBaron
            [ https://gitbox.apache.org/repos/asf?p=geode.git;h=e9f8563 ]

            GEODE-2542: Push DH keys when locator becomes coordinator

            This solves a race condition where a second locator (started at
            the same time) would fail to get a join response due to the join
            request being processed prior to becoming coordinator.

            Signed-off-by: Hitesh Khamesra <hiteshk25@yahoo.com>

            jira-bot ASF subversion and git services added a comment - Commit e9f856357565e72e8fe7dff7235404a74fdb5af9 in geode's branch refs/heads/develop from WireBaron [ https://gitbox.apache.org/repos/asf?p=geode.git;h=e9f8563 ] GEODE-2542 : Push DH keys when locator becomes coordinator This solves a race condition where a second locator (started at the same time) would fail to get a join response due to the join request being processed prior to becoming coordinator. Signed-off-by: Hitesh Khamesra <hiteshk25@yahoo.com>

            Commit e9f856357565e72e8fe7dff7235404a74fdb5af9 in geode's branch refs/heads/feature/GEODE-3026 from WireBaron
            [ https://gitbox.apache.org/repos/asf?p=geode.git;h=e9f8563 ]

            GEODE-2542: Push DH keys when locator becomes coordinator

            This solves a race condition where a second locator (started at
            the same time) would fail to get a join response due to the join
            request being processed prior to becoming coordinator.

            Signed-off-by: Hitesh Khamesra <hiteshk25@yahoo.com>

            jira-bot ASF subversion and git services added a comment - Commit e9f856357565e72e8fe7dff7235404a74fdb5af9 in geode's branch refs/heads/feature/ GEODE-3026 from WireBaron [ https://gitbox.apache.org/repos/asf?p=geode.git;h=e9f8563 ] GEODE-2542 : Push DH keys when locator becomes coordinator This solves a race condition where a second locator (started at the same time) would fail to get a join response due to the join request being processed prior to becoming coordinator. Signed-off-by: Hitesh Khamesra <hiteshk25@yahoo.com>

            Reopening as this is still failing in CI:

            org.apache.geode.distributed.LocatorUDPSecurityDUnitTest > testStartTwoLocators FAILED
                org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedCallable.call in VM 2 running on Host f2fdb491-3c55-462f-7cd7-57444e5c8493 with 5 VMs
            
                    Caused by:
                    org.apache.geode.InternalGemFireError: no distributed system found
            
                java.lang.AssertionError: Suspicious strings were written to the log during this run.
                Fix the strings or use IgnoredException.addIgnoredException to ignore.
                -----------------------------------------------------------------------
                Found suspect string in log4j at line 1300
            
                [error 2017/10/23 21:11:02.957 UTC <unicast receiver,f2fdb491-3c55-462f-7cd7-57444e5c8493-51452> tid=0x25c] Exception deserializing message payload: [dst: f2fdb491-3c55-462f-7cd7-57444e5c8493<v1>:32771, src: f2fdb491-3c55-462f-7cd7-57444e5c8493<v0>:32770 (2 headers), size=1452 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER]
                java.lang.Exception: Message id is -150
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1110)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1011)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1279)
                	at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
                	at org.jgroups.JChannel.up(JChannel.java:741)
                	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
                	at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
                	at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
                	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
                	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
                	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
                	at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
                	at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
                	at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
                	at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729)
                	at org.jgroups.protocols.TP.receive(TP.java:1654)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160)
                	at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
                	at java.lang.Thread.run(Thread.java:748)
                Caused by: java.lang.NullPointerException
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterSecretKey(GMSEncrypt.java:104)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterEncryptor(GMSEncrypt.java:290)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.decryptData(GMSEncrypt.java:153)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1089)
                	... 20 more
            
                -----------------------------------------------------------------------
                Found suspect string in log4j at line 1353
            
                [error 2017/10/23 21:11:12.938 UTC <unicast receiver,f2fdb491-3c55-462f-7cd7-57444e5c8493-51452> tid=0x25c] Exception deserializing message payload: [dst: f2fdb491-3c55-462f-7cd7-57444e5c8493<v1>:32771, src: f2fdb491-3c55-462f-7cd7-57444e5c8493<v0>:32770 (2 headers), size=234 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER]
                java.lang.Exception: Message id is -153
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1110)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1011)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1279)
                	at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
                	at org.jgroups.JChannel.up(JChannel.java:741)
                	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
                	at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
                	at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
                	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
                	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
                	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
                	at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
                	at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
                	at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
                	at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729)
                	at org.jgroups.protocols.TP.receive(TP.java:1654)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160)
                	at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
                	at java.lang.Thread.run(Thread.java:748)
                Caused by: java.lang.NullPointerException
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterSecretKey(GMSEncrypt.java:104)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterEncryptor(GMSEncrypt.java:290)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.decryptData(GMSEncrypt.java:153)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1089)
                	... 20 more
            
                -----------------------------------------------------------------------
                Found suspect string in log4j at line 1383
            
                [error 2017/10/23 21:11:12.939 UTC <unicast receiver,f2fdb491-3c55-462f-7cd7-57444e5c8493-51452> tid=0x25c] Exception deserializing message payload: [dst: f2fdb491-3c55-462f-7cd7-57444e5c8493<v1>:32771, src: f2fdb491-3c55-462f-7cd7-57444e5c8493<v0>:32770 (2 headers), size=924 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER]
                java.lang.Exception: Message id is -150
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1110)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1011)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1279)
                	at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
                	at org.jgroups.JChannel.up(JChannel.java:741)
                	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
                	at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
                	at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
                	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
                	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
                	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
                	at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
                	at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
                	at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
                	at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729)
                	at org.jgroups.protocols.TP.receive(TP.java:1654)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160)
                	at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
                	at java.lang.Thread.run(Thread.java:748)
                Caused by: java.lang.NullPointerException
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterSecretKey(GMSEncrypt.java:104)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterEncryptor(GMSEncrypt.java:290)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.decryptData(GMSEncrypt.java:153)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1089)
                	... 20 more
            
                -----------------------------------------------------------------------
                Found suspect string in log4j at line 1415
            
                [error 2017/10/23 21:11:12.940 UTC <unicast receiver,f2fdb491-3c55-462f-7cd7-57444e5c8493-51452> tid=0x25c] Exception deserializing message payload: [dst: f2fdb491-3c55-462f-7cd7-57444e5c8493<v1>:32771, src: f2fdb491-3c55-462f-7cd7-57444e5c8493<v0>:32770 (2 headers), size=924 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER]
                java.lang.Exception: Message id is -150
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1110)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1011)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1279)
                	at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
                	at org.jgroups.JChannel.up(JChannel.java:741)
                	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
                	at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
                	at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
                	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
                	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
                	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
                	at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
                	at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
                	at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
                	at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729)
                	at org.jgroups.protocols.TP.receive(TP.java:1654)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160)
                	at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
                	at java.lang.Thread.run(Thread.java:748)
                Caused by: java.lang.NullPointerException
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterSecretKey(GMSEncrypt.java:104)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterEncryptor(GMSEncrypt.java:290)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.decryptData(GMSEncrypt.java:153)
                	at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1089)
                	... 20 more
            
            gosullivan Galen O'Sullivan added a comment - Reopening as this is still failing in CI: org.apache.geode.distributed.LocatorUDPSecurityDUnitTest > testStartTwoLocators FAILED org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedCallable.call in VM 2 running on Host f2fdb491-3c55-462f-7cd7-57444e5c8493 with 5 VMs Caused by: org.apache.geode.InternalGemFireError: no distributed system found java.lang.AssertionError: Suspicious strings were written to the log during this run. Fix the strings or use IgnoredException.addIgnoredException to ignore. ----------------------------------------------------------------------- Found suspect string in log4j at line 1300 [error 2017/10/23 21:11:02.957 UTC <unicast receiver,f2fdb491-3c55-462f-7cd7-57444e5c8493-51452> tid=0x25c] Exception deserializing message payload: [dst: f2fdb491-3c55-462f-7cd7-57444e5c8493<v1>:32771, src: f2fdb491-3c55-462f-7cd7-57444e5c8493<v0>:32770 (2 headers), size=1452 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER] java.lang.Exception: Message id is -150 at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1110) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1011) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1279) at org.jgroups.JChannel.invokeCallback(JChannel.java:816) at org.jgroups.JChannel.up(JChannel.java:741) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030) at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:390) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426) at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74) at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72) at org.jgroups.protocols.TP.passMessageUp(TP.java:1601) at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817) at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10) at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729) at org.jgroups.protocols.TP.receive(TP.java:1654) at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160) at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701) at java.lang. Thread .run( Thread .java:748) Caused by: java.lang.NullPointerException at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterSecretKey(GMSEncrypt.java:104) at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterEncryptor(GMSEncrypt.java:290) at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.decryptData(GMSEncrypt.java:153) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1089) ... 20 more ----------------------------------------------------------------------- Found suspect string in log4j at line 1353 [error 2017/10/23 21:11:12.938 UTC <unicast receiver,f2fdb491-3c55-462f-7cd7-57444e5c8493-51452> tid=0x25c] Exception deserializing message payload: [dst: f2fdb491-3c55-462f-7cd7-57444e5c8493<v1>:32771, src: f2fdb491-3c55-462f-7cd7-57444e5c8493<v0>:32770 (2 headers), size=234 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER] java.lang.Exception: Message id is -153 at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1110) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1011) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1279) at org.jgroups.JChannel.invokeCallback(JChannel.java:816) at org.jgroups.JChannel.up(JChannel.java:741) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030) at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:390) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426) at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74) at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72) at org.jgroups.protocols.TP.passMessageUp(TP.java:1601) at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817) at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10) at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729) at org.jgroups.protocols.TP.receive(TP.java:1654) at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160) at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701) at java.lang. Thread .run( Thread .java:748) Caused by: java.lang.NullPointerException at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterSecretKey(GMSEncrypt.java:104) at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterEncryptor(GMSEncrypt.java:290) at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.decryptData(GMSEncrypt.java:153) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1089) ... 20 more ----------------------------------------------------------------------- Found suspect string in log4j at line 1383 [error 2017/10/23 21:11:12.939 UTC <unicast receiver,f2fdb491-3c55-462f-7cd7-57444e5c8493-51452> tid=0x25c] Exception deserializing message payload: [dst: f2fdb491-3c55-462f-7cd7-57444e5c8493<v1>:32771, src: f2fdb491-3c55-462f-7cd7-57444e5c8493<v0>:32770 (2 headers), size=924 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER] java.lang.Exception: Message id is -150 at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1110) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1011) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1279) at org.jgroups.JChannel.invokeCallback(JChannel.java:816) at org.jgroups.JChannel.up(JChannel.java:741) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030) at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:390) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426) at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74) at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72) at org.jgroups.protocols.TP.passMessageUp(TP.java:1601) at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817) at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10) at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729) at org.jgroups.protocols.TP.receive(TP.java:1654) at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160) at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701) at java.lang. Thread .run( Thread .java:748) Caused by: java.lang.NullPointerException at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterSecretKey(GMSEncrypt.java:104) at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterEncryptor(GMSEncrypt.java:290) at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.decryptData(GMSEncrypt.java:153) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1089) ... 20 more ----------------------------------------------------------------------- Found suspect string in log4j at line 1415 [error 2017/10/23 21:11:12.940 UTC <unicast receiver,f2fdb491-3c55-462f-7cd7-57444e5c8493-51452> tid=0x25c] Exception deserializing message payload: [dst: f2fdb491-3c55-462f-7cd7-57444e5c8493<v1>:32771, src: f2fdb491-3c55-462f-7cd7-57444e5c8493<v0>:32770 (2 headers), size=924 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER] java.lang.Exception: Message id is -150 at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1110) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1011) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1279) at org.jgroups.JChannel.invokeCallback(JChannel.java:816) at org.jgroups.JChannel.up(JChannel.java:741) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030) at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:390) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426) at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74) at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72) at org.jgroups.protocols.TP.passMessageUp(TP.java:1601) at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817) at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10) at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729) at org.jgroups.protocols.TP.receive(TP.java:1654) at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160) at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701) at java.lang. Thread .run( Thread .java:748) Caused by: java.lang.NullPointerException at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterSecretKey(GMSEncrypt.java:104) at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.getClusterEncryptor(GMSEncrypt.java:290) at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.decryptData(GMSEncrypt.java:153) at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1089) ... 20 more
            gosullivan Galen O'Sullivan added a comment - Failed again today in https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/FlakyTest/builds/182
            pivotalsarge Michael W. Dodge added a comment - - edited

            The last stack trace in the comments no longer appears to be possible:

            protected byte[] getClusterSecretKey() {
                if (this.clusterEncryptor != null) {
                  return this.clusterEncryptor.getSecretBytes();
                } else {
                  return null;
                }
              }
            
            
            pivotalsarge Michael W. Dodge added a comment - - edited The last stack trace in the comments no longer appears to be possible: protected byte[] getClusterSecretKey() { if (this.clusterEncryptor != null) { return this.clusterEncryptor.getSecretBytes(); } else { return null; } }
            pivotalsarge Michael W. Dodge added a comment - - edited

            testStartTwoLocators includes a 10-second wait for the view to have five members. Is this test flaky simply because under some conditions it takes longer than 10 seconds without there being an error condition, e.g., one of the machines is very, very slow? After running it 700 times locally there were zero failures.

            pivotalsarge Michael W. Dodge added a comment - - edited testStartTwoLocators includes a 10-second wait for the view to have five members. Is this test flaky simply because under some conditions it takes longer than 10 seconds without there being an error condition, e.g., one of the machines is very, very slow? After running it 700 times locally there were zero failures.

            Commit 91ed05a3d6b195a2f361a166b6c97a745bbb0f59 in geode's branch refs/heads/develop from gosullivan
            [ https://gitbox.apache.org/repos/asf?p=geode.git;h=91ed05a ]

            GEODE-2542: Extend timeouts from 10s to 30; cleanup (#1897)

            GEODE-2542: Extend timeouts from 10s to 30; cleanup

            • Remove usage of Assert.fail()
            • Don't use invoke try-catch-throw paradigm.
            • Remove dead code
            • Un-ignore tests, use implicit default constructors.
            jira-bot ASF subversion and git services added a comment - Commit 91ed05a3d6b195a2f361a166b6c97a745bbb0f59 in geode's branch refs/heads/develop from gosullivan [ https://gitbox.apache.org/repos/asf?p=geode.git;h=91ed05a ] GEODE-2542 : Extend timeouts from 10s to 30; cleanup (#1897) GEODE-2542 : Extend timeouts from 10s to 30; cleanup Remove usage of Assert.fail() Don't use invoke try-catch-throw paradigm. Remove dead code Un-ignore tests, use implicit default constructors.

            Commit 91ed05a3d6b195a2f361a166b6c97a745bbb0f59 in geode's branch refs/heads/develop from gosullivan
            [ https://gitbox.apache.org/repos/asf?p=geode.git;h=91ed05a ]

            GEODE-2542: Extend timeouts from 10s to 30; cleanup (#1897)

            GEODE-2542: Extend timeouts from 10s to 30; cleanup

            • Remove usage of Assert.fail()
            • Don't use invoke try-catch-throw paradigm.
            • Remove dead code
            • Un-ignore tests, use implicit default constructors.
            jira-bot ASF subversion and git services added a comment - Commit 91ed05a3d6b195a2f361a166b6c97a745bbb0f59 in geode's branch refs/heads/develop from gosullivan [ https://gitbox.apache.org/repos/asf?p=geode.git;h=91ed05a ] GEODE-2542 : Extend timeouts from 10s to 30; cleanup (#1897) GEODE-2542 : Extend timeouts from 10s to 30; cleanup Remove usage of Assert.fail() Don't use invoke try-catch-throw paradigm. Remove dead code Un-ignore tests, use implicit default constructors.

            I bumped up the timeouts from 10s to 30s and did some test cleanup. Hopefully this is enough to fix the flakiness.

            gosullivan Galen O'Sullivan added a comment - I bumped up the timeouts from 10s to 30s and did some test cleanup. Hopefully this is enough to fix the flakiness.

            Commit 91ed05a3d6b195a2f361a166b6c97a745bbb0f59 in geode's branch refs/heads/feature/GEODE-5166 from gosullivan
            [ https://gitbox.apache.org/repos/asf?p=geode.git;h=91ed05a ]

            GEODE-2542: Extend timeouts from 10s to 30; cleanup (#1897)

            GEODE-2542: Extend timeouts from 10s to 30; cleanup

            • Remove usage of Assert.fail()
            • Don't use invoke try-catch-throw paradigm.
            • Remove dead code
            • Un-ignore tests, use implicit default constructors.
            jira-bot ASF subversion and git services added a comment - Commit 91ed05a3d6b195a2f361a166b6c97a745bbb0f59 in geode's branch refs/heads/feature/ GEODE-5166 from gosullivan [ https://gitbox.apache.org/repos/asf?p=geode.git;h=91ed05a ] GEODE-2542 : Extend timeouts from 10s to 30; cleanup (#1897) GEODE-2542 : Extend timeouts from 10s to 30; cleanup Remove usage of Assert.fail() Don't use invoke try-catch-throw paradigm. Remove dead code Un-ignore tests, use implicit default constructors.

            Commit 91ed05a3d6b195a2f361a166b6c97a745bbb0f59 in geode's branch refs/heads/feature/GEODE-5166 from gosullivan
            [ https://gitbox.apache.org/repos/asf?p=geode.git;h=91ed05a ]

            GEODE-2542: Extend timeouts from 10s to 30; cleanup (#1897)

            GEODE-2542: Extend timeouts from 10s to 30; cleanup

            • Remove usage of Assert.fail()
            • Don't use invoke try-catch-throw paradigm.
            • Remove dead code
            • Un-ignore tests, use implicit default constructors.
            jira-bot ASF subversion and git services added a comment - Commit 91ed05a3d6b195a2f361a166b6c97a745bbb0f59 in geode's branch refs/heads/feature/ GEODE-5166 from gosullivan [ https://gitbox.apache.org/repos/asf?p=geode.git;h=91ed05a ] GEODE-2542 : Extend timeouts from 10s to 30; cleanup (#1897) GEODE-2542 : Extend timeouts from 10s to 30; cleanup Remove usage of Assert.fail() Don't use invoke try-catch-throw paradigm. Remove dead code Un-ignore tests, use implicit default constructors.

            People

              gosullivan Galen O'Sullivan
              klund Kirk Lund
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m