Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-1793

Flaky: LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL

Details

    • Bug
    • Status: Reopened
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • locator, membership
    • None

    Description

      This test fails due to something not cleaning itself properly. Undetermined what the problem is, but it will run perfectly by itself everytime, but once run inside of the TestClass it fails

      Attachments

        Issue Links

          Activity

            fixed in dunit cleanup method

            bschuchardt Bruce J Schuchardt added a comment - fixed in dunit cleanup method

            This test failed during a CI run

            org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedRunnable.run in VM 2 running on Host cc4-rh6.gemstone.com with 4 VMs
            	at org.apache.geode.test.dunit.VM.invoke(VM.java:389)
            	at org.apache.geode.test.dunit.VM.invoke(VM.java:355)
            	at org.apache.geode.test.dunit.VM.invoke(VM.java:280)
            	at org.apache.geode.distributed.LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL(LocatorDUnitTest.java:481)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:497)
            	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
            	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
            	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
            	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
            	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
            	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
            	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
            	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
            	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
            	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
            	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
            	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
            	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
            	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
            	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
            	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
            	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
            	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
            	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
            	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
            	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
            	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:497)
            	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
            	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
            	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
            	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
            	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
            	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:497)
            	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
            	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
            	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377)
            	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
            	at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            	at java.lang.Thread.run(Thread.java:745)
            Caused by: org.apache.geode.SystemConnectException: Problem starting up membership services
            	at org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:116)
            	at org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:92)
            	at org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1092)
            	at org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1144)
            	at org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:521)
            	at org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:659)
            	at org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)
            	at org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:237)
            	at org.apache.geode.distributed.internal.InternalLocator.startDistributedSystem(InternalLocator.java:746)
            	at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:395)
            	at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:336)
            	at org.apache.geode.distributed.Locator.startLocator(Locator.java:334)
            	at org.apache.geode.distributed.Locator.startLocatorAndDS(Locator.java:166)
            	at org.apache.geode.distributed.LocatorDUnitTest.startLocator(LocatorDUnitTest.java:627)
            	at org.apache.geode.distributed.LocatorDUnitTest.lambda$testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL$1965b9f9$1(LocatorDUnitTest.java:481)
            	at org.apache.geode.test.dunit.NamedRunnable.run(NamedRunnable.java:33)
            	at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:497)
            	at hydra.MethExecutor.executeObject(MethExecutor.java:268)
            	at org.apache.geode.test.dunit.standalone.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:82)
            	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:497)
            	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
            	at sun.rmi.transport.Transport$1.run(Transport.java:200)
            	at sun.rmi.transport.Transport$1.run(Transport.java:197)
            	at java.security.AccessController.doPrivileged(Native Method)
            	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
            	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
            	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
            	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$95(TCPTransport.java:683)
            	at java.security.AccessController.doPrivileged(Native Method)
            	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
            	... 3 more
            Caused by: org.apache.geode.distributed.internal.tcpserver.LocatorCancelException
            	at org.apache.geode.distributed.internal.tcpserver.TcpClient.getServerVersion(TcpClient.java:272)
            	at org.apache.geode.distributed.internal.tcpserver.TcpClient.requestToServer(TcpClient.java:157)
            	at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$TcpClientWrapper.sendCoordinatorFindRequest(GMSJoinLeave.java:1122)
            	at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.findCoordinator(GMSJoinLeave.java:1037)
            	at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:302)
            	at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:662)
            	at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:749)
            	at org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:183)
            	at org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:104)
            	... 36 more
            
            bschuchardt Bruce J Schuchardt added a comment - This test failed during a CI run org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedRunnable.run in VM 2 running on Host cc4-rh6.gemstone.com with 4 VMs at org.apache.geode.test.dunit.VM.invoke(VM.java:389) at org.apache.geode.test.dunit.VM.invoke(VM.java:355) at org.apache.geode.test.dunit.VM.invoke(VM.java:280) at org.apache.geode.distributed.LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL(LocatorDUnitTest.java:481) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54) at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.geode.SystemConnectException: Problem starting up membership services at org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:116) at org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:92) at org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1092) at org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1144) at org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:521) at org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:659) at org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299) at org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:237) at org.apache.geode.distributed.internal.InternalLocator.startDistributedSystem(InternalLocator.java:746) at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:395) at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:336) at org.apache.geode.distributed.Locator.startLocator(Locator.java:334) at org.apache.geode.distributed.Locator.startLocatorAndDS(Locator.java:166) at org.apache.geode.distributed.LocatorDUnitTest.startLocator(LocatorDUnitTest.java:627) at org.apache.geode.distributed.LocatorDUnitTest.lambda$testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL$1965b9f9$1(LocatorDUnitTest.java:481) at org.apache.geode.test.dunit.NamedRunnable.run(NamedRunnable.java:33) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at hydra.MethExecutor.executeObject(MethExecutor.java:268) at org.apache.geode.test.dunit.standalone.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:82) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$95(TCPTransport.java:683) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) ... 3 more Caused by: org.apache.geode.distributed.internal.tcpserver.LocatorCancelException at org.apache.geode.distributed.internal.tcpserver.TcpClient.getServerVersion(TcpClient.java:272) at org.apache.geode.distributed.internal.tcpserver.TcpClient.requestToServer(TcpClient.java:157) at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$TcpClientWrapper.sendCoordinatorFindRequest(GMSJoinLeave.java:1122) at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.findCoordinator(GMSJoinLeave.java:1037) at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:302) at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:662) at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:749) at org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:183) at org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:104) ... 36 more

            this ticket describes the test as Flaky but the test has not been annotated as being flaky and so is still being run with regular unit tests

            bschuchardt Bruce J Schuchardt added a comment - this ticket describes the test as Flaky but the test has not been annotated as being flaky and so is still being run with regular unit tests

            Sometimes, there is a RMIException thrown to the actual caller as a result of the connect failing, rather than it happening asynchronously. I haven't gone deep enough into the locator to find the reason for this, but a fix for the test is to catch the exception before it propagates as an RMIException.

            gosullivan Galen O'Sullivan added a comment - Sometimes, there is a RMIException thrown to the actual caller as a result of the connect failing, rather than it happening asynchronously. I haven't gone deep enough into the locator to find the reason for this, but a fix for the test is to catch the exception before it propagates as an RMIException.
            gosullivan Galen O'Sullivan added a comment - - edited

            Of note: the Locator makes two socket connections, one to get the header, and then one to get the rest of the connection. I feel that this could be better if we didn't make so many connections, but that would break backward compatibility.

            gosullivan Galen O'Sullivan added a comment - - edited Of note: the Locator makes two socket connections, one to get the header, and then one to get the rest of the connection. I feel that this could be better if we didn't make so many connections, but that would break backward compatibility.
            githubbot ASF GitHub Bot added a comment -

            GitHub user galen-pivotal opened a pull request:

            https://github.com/apache/geode/pull/360

            GEODE-1793 Fix a flaky test, clean up SSL tests.

            • Use less code.
            • Get random port numbers a bit more robustly.
            • Check to make sure that locator2 has died at the end.

            You can merge this pull request into a Git repository by running:

            $ git pull https://github.com/galen-pivotal/incubator-geode feature/GEODE-1793

            Alternatively you can review and apply these changes as the patch at:

            https://github.com/apache/geode/pull/360.patch

            To close this pull request, make a commit to your master/trunk branch
            with (at least) the following in the commit message:

            This closes #360


            commit b5a9f347b73cdbc0a5e401db5e7b11447a062045
            Author: Galen O'Sullivan <gosullivan@pivotal.io>
            Date: 2017-01-11T03:56:42Z

            GEODE-1793 Fix a flaky test, clean up SSL tests.

            • Use less code.
            • Get random port numbers a bit more robustly.
            • Check to make sure that locator2 has died at the end.

            githubbot ASF GitHub Bot added a comment - GitHub user galen-pivotal opened a pull request: https://github.com/apache/geode/pull/360 GEODE-1793 Fix a flaky test, clean up SSL tests. Use less code. Get random port numbers a bit more robustly. Check to make sure that locator2 has died at the end. You can merge this pull request into a Git repository by running: $ git pull https://github.com/galen-pivotal/incubator-geode feature/ GEODE-1793 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/geode/pull/360.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #360 commit b5a9f347b73cdbc0a5e401db5e7b11447a062045 Author: Galen O'Sullivan <gosullivan@pivotal.io> Date: 2017-01-11T03:56:42Z GEODE-1793 Fix a flaky test, clean up SSL tests. Use less code. Get random port numbers a bit more robustly. Check to make sure that locator2 has died at the end.

            Commit 8426c675c464896eab659a86fdb7a8c17706c607 in geode's branch refs/heads/develop from gosullivan
            [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=8426c67 ]

            GEODE-1793: Fix a flaky test, clean up SSL tests. This closes #360

            • Add a couple of common test methods.
            • Get random port numbers a bit more robustly.
            • Check to make sure that locator2 has died at the end.
            jira-bot ASF subversion and git services added a comment - Commit 8426c675c464896eab659a86fdb7a8c17706c607 in geode's branch refs/heads/develop from gosullivan [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=8426c67 ] GEODE-1793 : Fix a flaky test, clean up SSL tests. This closes #360 Add a couple of common test methods. Get random port numbers a bit more robustly. Check to make sure that locator2 has died at the end.
            githubbot ASF GitHub Bot added a comment -

            Github user asfgit closed the pull request at:

            https://github.com/apache/geode/pull/360

            githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/geode/pull/360

            I managed to get this test to fail & see that the second locator, in vm_2, was supposed to shut down but instead it created its own cluster:

            [vm2] [info 2017/03/02 15:19:56.969 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Starting membership services
            
            [vm2] [info 2017/03/02 15:19:56.979 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] JGroups channel created (took 9ms)
            
            [vm2] [info 2017/03/02 15:19:56.980 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] GemFire P2P Listener started on /10.118.32.92:39824
            
            [vm2] [info 2017/03/02 15:19:56.980 PST <Geode Failure Detection Server thread 0> tid=0x21c] Started failure detection server thread on /10.118.32.92:59150.
            
            [vm2] [info 2017/03/02 15:19:56.980 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Peer locator is connecting to local membership services with ID trout(2830:locator)<ec>:32771
            
            [vm2] [info 2017/03/02 15:19:57.164 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] This member is becoming the membership coordinator with address trout(2830:locator)<ec>:32771
            

            It was correctly configured with two locator addresses and its SSL configuration appeared to be okay. The other locator, in vm_1, correctly had SSL disabled. The SSL locator was unable to recover from the non-SSL locator:

            [vm2] [info 2017/03/02 15:19:56.961 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] GemFire peer location service starting.  Other locators: trout.gemstone.com[46843]  Locators preferred as coordinators: false  Network partition detection enabled: false  View persistence file: locator0view.dat
            
            [vm2] [info 2017/03/02 15:19:56.961 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Peer locator attempting to recover from trout.gemstone.com/10.118.32.92:46843
            
            [vm2] [info 2017/03/02 15:19:56.963 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Peer locator was unable to recover state from this locator
            
            [vm2] [info 2017/03/02 15:19:56.963 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] recovery file not found: /export/trout1/users/bschuchardt/devel/gfdev/open/geode-core/build/distributedTest/dunit/vm2/locator0view.dat
            
            [vm2] [info 2017/03/02 15:19:56.963 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Starting distributed system
            

            So this appears to be a problem with Geode, not the test. It is the Geode functionality that is "flaky", working most of the time but not 100%.

            bschuchardt Bruce J Schuchardt added a comment - I managed to get this test to fail & see that the second locator, in vm_2, was supposed to shut down but instead it created its own cluster: [vm2] [info 2017/03/02 15:19:56.969 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Starting membership services [vm2] [info 2017/03/02 15:19:56.979 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] JGroups channel created (took 9ms) [vm2] [info 2017/03/02 15:19:56.980 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] GemFire P2P Listener started on /10.118.32.92:39824 [vm2] [info 2017/03/02 15:19:56.980 PST <Geode Failure Detection Server thread 0> tid=0x21c] Started failure detection server thread on /10.118.32.92:59150. [vm2] [info 2017/03/02 15:19:56.980 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Peer locator is connecting to local membership services with ID trout(2830:locator)<ec>:32771 [vm2] [info 2017/03/02 15:19:57.164 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] This member is becoming the membership coordinator with address trout(2830:locator)<ec>:32771 It was correctly configured with two locator addresses and its SSL configuration appeared to be okay. The other locator, in vm_1, correctly had SSL disabled. The SSL locator was unable to recover from the non-SSL locator: [vm2] [info 2017/03/02 15:19:56.961 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] GemFire peer location service starting. Other locators: trout.gemstone.com[46843] Locators preferred as coordinators: false Network partition detection enabled: false View persistence file: locator0view.dat [vm2] [info 2017/03/02 15:19:56.961 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Peer locator attempting to recover from trout.gemstone.com/10.118.32.92:46843 [vm2] [info 2017/03/02 15:19:56.963 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Peer locator was unable to recover state from this locator [vm2] [info 2017/03/02 15:19:56.963 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] recovery file not found: /export/trout1/users/bschuchardt/devel/gfdev/open/geode-core/build/distributedTest/dunit/vm2/locator0view.dat [vm2] [info 2017/03/02 15:19:56.963 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Starting distributed system So this appears to be a problem with Geode, not the test. It is the Geode functionality that is "flaky", working most of the time but not 100%.

            In another run with modified logging I can see that the SSL locator did not see an SSLHandshakeException as expected but merely an IOException

            [vm2] [info 2017/03/02 15:40:00.701 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Peer locator could not recover membership view from trout.gemstone.com/10.118.32.92:38254: Connection reset

            So this appears to be an inconsistency in SSL implementations used by the JVM.

            bschuchardt Bruce J Schuchardt added a comment - In another run with modified logging I can see that the SSL locator did not see an SSLHandshakeException as expected but merely an IOException [vm2] [info 2017/03/02 15:40:00.701 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Peer locator could not recover membership view from trout.gemstone.com/10.118.32.92:38254: Connection reset So this appears to be an inconsistency in SSL implementations used by the JVM.

            Commit 866dc5ca1583c5fab49ec96c48d261c0367427f3 in geode's branch refs/heads/feature/GEODE-1793 from bschuchardt
            [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=866dc5c ]

            GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL

            This was a product issue. When the locator using plain-text sockets is
            contacted by a TcpClient using SSL the locator often just closes the socket.
            On some platforms this causes a SSLHandshakeException but on others it
            just causes a "SocketException: connection reset". Writing some text to
            the socket forces the TcpClient to get a SSLException (which is the superclass
            of SSLHandshakeException).

            The test class is still marked as Flaky due to GEODE-2542.

            jira-bot ASF subversion and git services added a comment - Commit 866dc5ca1583c5fab49ec96c48d261c0367427f3 in geode's branch refs/heads/feature/ GEODE-1793 from bschuchardt [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=866dc5c ] GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL This was a product issue. When the locator using plain-text sockets is contacted by a TcpClient using SSL the locator often just closes the socket. On some platforms this causes a SSLHandshakeException but on others it just causes a "SocketException: connection reset". Writing some text to the socket forces the TcpClient to get a SSLException (which is the superclass of SSLHandshakeException). The test class is still marked as Flaky due to GEODE-2542 .
            githubbot ASF GitHub Bot added a comment -

            GitHub user bschuchardt opened a pull request:

            https://github.com/apache/geode/pull/412

            GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOther…

            This was a product issue. When the locator using plain-text sockets is
            contacted by a TcpClient using SSL the locator often just closes the socket.
            On some platforms this causes a SSLHandshakeException but on others it
            just causes a "SocketException: connection reset". Writing some text to
            the socket forces the TcpClient to get a SSLException (which is the superclass
            of SSLHandshakeException).

            The test class is still marked as Flaky due to GEODE-2542.

            I deleted one of the tests in LocatorDUnitTest as it wasn't doing any useful validation and really served no purpose.

            I also increased the joinTimeout in this test. The original 1-second timeout was intended to make the tests run faster but I think it's probably the source of some of the flaky-ness in this set of tests. Some of them were also overriding the joinTimeout established by the DUnit framework, so that was actually a bad thing to be doing. The tests all run in a few seconds with the default joinTimeout setting anyway.

            You can merge this pull request into a Git repository by running:

            $ git pull https://github.com/apache/geode feature/GEODE-1793

            Alternatively you can review and apply these changes as the patch at:

            https://github.com/apache/geode/pull/412.patch

            To close this pull request, make a commit to your master/trunk branch
            with (at least) the following in the commit message:

            This closes #412


            commit 866dc5ca1583c5fab49ec96c48d261c0367427f3
            Author: Bruce Schuchardt <bschuchardt@pivotal.io>
            Date: 2017-03-03T21:47:42Z

            GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL

            This was a product issue. When the locator using plain-text sockets is
            contacted by a TcpClient using SSL the locator often just closes the socket.
            On some platforms this causes a SSLHandshakeException but on others it
            just causes a "SocketException: connection reset". Writing some text to
            the socket forces the TcpClient to get a SSLException (which is the superclass
            of SSLHandshakeException).

            The test class is still marked as Flaky due to GEODE-2542.


            githubbot ASF GitHub Bot added a comment - GitHub user bschuchardt opened a pull request: https://github.com/apache/geode/pull/412 GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOther… This was a product issue. When the locator using plain-text sockets is contacted by a TcpClient using SSL the locator often just closes the socket. On some platforms this causes a SSLHandshakeException but on others it just causes a "SocketException: connection reset". Writing some text to the socket forces the TcpClient to get a SSLException (which is the superclass of SSLHandshakeException). The test class is still marked as Flaky due to GEODE-2542 . I deleted one of the tests in LocatorDUnitTest as it wasn't doing any useful validation and really served no purpose. I also increased the joinTimeout in this test. The original 1-second timeout was intended to make the tests run faster but I think it's probably the source of some of the flaky-ness in this set of tests. Some of them were also overriding the joinTimeout established by the DUnit framework, so that was actually a bad thing to be doing. The tests all run in a few seconds with the default joinTimeout setting anyway. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/geode feature/ GEODE-1793 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/geode/pull/412.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #412 commit 866dc5ca1583c5fab49ec96c48d261c0367427f3 Author: Bruce Schuchardt <bschuchardt@pivotal.io> Date: 2017-03-03T21:47:42Z GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL This was a product issue. When the locator using plain-text sockets is contacted by a TcpClient using SSL the locator often just closes the socket. On some platforms this causes a SSLHandshakeException but on others it just causes a "SocketException: connection reset". Writing some text to the socket forces the TcpClient to get a SSLException (which is the superclass of SSLHandshakeException). The test class is still marked as Flaky due to GEODE-2542 .
            githubbot ASF GitHub Bot added a comment -

            Github user bschuchardt commented on the issue:

            https://github.com/apache/geode/pull/412

            There are "spotless" problems I'm cleaning up, and I'm removing the commented out code from the test.

            githubbot ASF GitHub Bot added a comment - Github user bschuchardt commented on the issue: https://github.com/apache/geode/pull/412 There are "spotless" problems I'm cleaning up, and I'm removing the commented out code from the test.
            githubbot ASF GitHub Bot added a comment -

            Github user galen-pivotal commented on a diff in the pull request:

            https://github.com/apache/geode/pull/412#discussion_r104258182

            — Diff: geode-core/src/main/java/org/apache/geode/distributed/internal/tcpserver/TcpServer.java —
            @@ -360,6 +360,13 @@ private void processRequest(final Socket sock)

            { versionOrdinal = (short) GOSSIP_TO_GEMFIRE_VERSION_MAP.get(gossipVersion); }

            else {
            // Close the socket. We can not accept requests from a newer version
            + try {
            + sock.getOutputStream().write("unknown protocol version".getBytes());
            — End diff –

            Is there any risk of this being interpreted by anything other than garbage on the other side?

            githubbot ASF GitHub Bot added a comment - Github user galen-pivotal commented on a diff in the pull request: https://github.com/apache/geode/pull/412#discussion_r104258182 — Diff: geode-core/src/main/java/org/apache/geode/distributed/internal/tcpserver/TcpServer.java — @@ -360,6 +360,13 @@ private void processRequest(final Socket sock) { versionOrdinal = (short) GOSSIP_TO_GEMFIRE_VERSION_MAP.get(gossipVersion); } else { // Close the socket. We can not accept requests from a newer version + try { + sock.getOutputStream().write("unknown protocol version".getBytes()); — End diff – Is there any risk of this being interpreted by anything other than garbage on the other side?
            githubbot ASF GitHub Bot added a comment -

            Github user galen-pivotal commented on a diff in the pull request:

            https://github.com/apache/geode/pull/412#discussion_r104258974

            — Diff: geode-core/src/main/java/org/apache/geode/distributed/internal/tcpserver/TcpServer.java —
            @@ -77,9 +77,9 @@

            • <p>
            • This should be incremented if the gossip message structures change
            • <p>
            • * 1000 - gemfire 5.5 - using java serialization 1001 - 5.7 - using DataSerializable and
                • End diff –

            Did spotless break this? 😮

            githubbot ASF GitHub Bot added a comment - Github user galen-pivotal commented on a diff in the pull request: https://github.com/apache/geode/pull/412#discussion_r104258974 — Diff: geode-core/src/main/java/org/apache/geode/distributed/internal/tcpserver/TcpServer.java — @@ -77,9 +77,9 @@ <p> This should be incremented if the gossip message structures change <p> * 1000 - gemfire 5.5 - using java serialization 1001 - 5.7 - using DataSerializable and End diff – Did spotless break this? 😮
            githubbot ASF GitHub Bot added a comment -

            Github user bschuchardt commented on a diff in the pull request:

            https://github.com/apache/geode/pull/412#discussion_r104259153

            — Diff: geode-core/src/main/java/org/apache/geode/distributed/internal/tcpserver/TcpServer.java —
            @@ -77,9 +77,9 @@

            • <p>
            • This should be incremented if the gossip message structures change
            • <p>
            • * 1000 - gemfire 5.5 - using java serialization 1001 - 5.7 - using DataSerializable and
                • End diff –

            I didn't check but that's my assumption. It likes the comment with the HTML breaks in place.

            githubbot ASF GitHub Bot added a comment - Github user bschuchardt commented on a diff in the pull request: https://github.com/apache/geode/pull/412#discussion_r104259153 — Diff: geode-core/src/main/java/org/apache/geode/distributed/internal/tcpserver/TcpServer.java — @@ -77,9 +77,9 @@ <p> This should be incremented if the gossip message structures change <p> * 1000 - gemfire 5.5 - using java serialization 1001 - 5.7 - using DataSerializable and End diff – I didn't check but that's my assumption. It likes the comment with the HTML breaks in place.
            githubbot ASF GitHub Bot added a comment -

            Github user bschuchardt commented on a diff in the pull request:

            https://github.com/apache/geode/pull/412#discussion_r104260160

            — Diff: geode-core/src/main/java/org/apache/geode/distributed/internal/tcpserver/TcpServer.java —
            @@ -360,6 +360,13 @@ private void processRequest(final Socket sock)

            { versionOrdinal = (short) GOSSIP_TO_GEMFIRE_VERSION_MAP.get(gossipVersion); }

            else {
            // Close the socket. We can not accept requests from a newer version
            + try {
            + sock.getOutputStream().write("unknown protocol version".getBytes());
            — End diff –

            This situation should never happen. When we fetch the protocol version of a Locator we use v5.7. Then we use the locator's protocol version in future communications. We will never send a higher version number than the locator knows how to handle.

            For SSL, this text is nothing like the response to a HelloClient packet. On machines I've tested on it results in either a SSLHandshakeException or a SSLException hinting that the other end might be using plain text sockets.

            githubbot ASF GitHub Bot added a comment - Github user bschuchardt commented on a diff in the pull request: https://github.com/apache/geode/pull/412#discussion_r104260160 — Diff: geode-core/src/main/java/org/apache/geode/distributed/internal/tcpserver/TcpServer.java — @@ -360,6 +360,13 @@ private void processRequest(final Socket sock) { versionOrdinal = (short) GOSSIP_TO_GEMFIRE_VERSION_MAP.get(gossipVersion); } else { // Close the socket. We can not accept requests from a newer version + try { + sock.getOutputStream().write("unknown protocol version".getBytes()); — End diff – This situation should never happen. When we fetch the protocol version of a Locator we use v5.7. Then we use the locator's protocol version in future communications. We will never send a higher version number than the locator knows how to handle. For SSL, this text is nothing like the response to a HelloClient packet. On machines I've tested on it results in either a SSLHandshakeException or a SSLException hinting that the other end might be using plain text sockets.
            githubbot ASF GitHub Bot added a comment -

            Github user bschuchardt commented on the issue:

            https://github.com/apache/geode/pull/412

            remote: geode git commit: GEODE_1793 spotless fixes and removal of dead code
            remote: geode git commit: GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL
            To https://git-wip-us.apache.org/repos/asf/geode.git
            c09a856..4112204 develop -> develop

            githubbot ASF GitHub Bot added a comment - Github user bschuchardt commented on the issue: https://github.com/apache/geode/pull/412 remote: geode git commit: GEODE_1793 spotless fixes and removal of dead code remote: geode git commit: GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL To https://git-wip-us.apache.org/repos/asf/geode.git c09a856..4112204 develop -> develop
            githubbot ASF GitHub Bot added a comment -

            Github user bschuchardt closed the pull request at:

            https://github.com/apache/geode/pull/412

            githubbot ASF GitHub Bot added a comment - Github user bschuchardt closed the pull request at: https://github.com/apache/geode/pull/412

            Commit 064362e90e150ad4ef5b269fab78f0cf2d6e5f4f in geode's branch refs/heads/develop from bschuchardt
            [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=064362e ]

            GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL

            This was a product issue. When the locator using plain-text sockets is
            contacted by a TcpClient using SSL the locator often just closes the socket.
            On some platforms this causes a SSLHandshakeException but on others it
            just causes a "SocketException: connection reset". Writing some text to
            the socket forces the TcpClient to get a SSLException (which is the superclass
            of SSLHandshakeException).

            The test class is still marked as Flaky due to GEODE-2542.

            jira-bot ASF subversion and git services added a comment - Commit 064362e90e150ad4ef5b269fab78f0cf2d6e5f4f in geode's branch refs/heads/develop from bschuchardt [ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=064362e ] GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL This was a product issue. When the locator using plain-text sockets is contacted by a TcpClient using SSL the locator often just closes the socket. On some platforms this causes a SSLHandshakeException but on others it just causes a "SocketException: connection reset". Writing some text to the socket forces the TcpClient to get a SSLException (which is the superclass of SSLHandshakeException). The test class is still marked as Flaky due to GEODE-2542 .
            klund Kirk Lund added a comment -

            Failing intermittently with:

            java.io.EOFException: Locator at HostAndPort[d5676dfc4e21:24193] did not respond. This is normal if the locator was shutdown. If it wasn't check its log for exceptions.
                  at org.apache.geode.distributed.internal.tcpserver.TcpClient.requestToServer(TcpClient.java:202)
                  at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.findCoordinator(GMSJoinLeave.java:1145)
                  at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:331)
                  at org.apache.geode.distributed.internal.membership.gms.GMSMembership.join(GMSMembership.java:573)
                  at org.apache.geode.distributed.internal.membership.gms.GMSMembership.access$1300(GMSMembership.java:71)
                  at org.apache.geode.distributed.internal.membership.gms.GMSMembership$ManagerImpl.joinDistributedSystem(GMSMembership.java:1972)
                  at org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:242)
                  at org.apache.geode.distributed.internal.membership.gms.GMSMembership.start(GMSMembership.java:1851)
                  at org.apache.geode.distributed.internal.DistributionImpl.start(DistributionImpl.java:171)
                  at org.apache.geode.distributed.internal.DistributionImpl.createDistribution(DistributionImpl.java:222)
                  at org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:464)
                  at org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:497)
                  at org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:326)
                  at org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:779)
                  at org.apache.geode.distributed.internal.InternalDistributedSystem.access$200(InternalDistributedSystem.java:135)
                  at org.apache.geode.distributed.internal.InternalDistributedSystem$Builder.build(InternalDistributedSystem.java:3033)
                  at org.apache.geode.distributed.internal.InternalDistributedSystem.connectInternal(InternalDistributedSystem.java:290)
                  at org.apache.geode.distributed.internal.InternalLocator.startDistributedSystem(InternalLocator.java:743)
                  at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:388)
                  at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:338)
                  at org.apache.geode.distributed.Locator.startLocator(Locator.java:252)
                  at org.apache.geode.distributed.Locator.startLocatorAndDS(Locator.java:139)
                  at org.apache.geode.distributed.LocatorDUnitTest.startLocatorWithPortAndProperties(LocatorDUnitTest.java:1507)
                  at org.apache.geode.distributed.LocatorDUnitTest.lambda$startVerifyAndStopLocator$e21733f2$1(LocatorDUnitTest.java:1539)
                  at org.apache.geode.test.dunit.internal.IdentifiableRunnable.run(IdentifiableRunnable.java:41)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:498)
                  at org.apache.geode.test.dunit.internal.MethodInvoker.executeObject(MethodInvoker.java:123)
                  at org.apache.geode.test.dunit.internal.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:78)
                  at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:498)
                  at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
                  at sun.rmi.transport.Transport$1.run(Transport.java:200)
                  at sun.rmi.transport.Transport$1.run(Transport.java:197)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
                  at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
                  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
                  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                  at java.lang.Thread.run(Thread.java:748)
                Caused by: java.io.EOFException
                  at java.io.DataInputStream.readByte(DataInputStream.java:267)
                  at org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2493)
                  at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2864)
                  at org.apache.geode.internal.InternalDataSerializer$1.readObject(InternalDataSerializer.java:301)
                  at org.apache.geode.distributed.internal.tcpserver.TcpClient.requestToServer(TcpClient.java:197)
                  ... 46 more
            
            klund Kirk Lund added a comment - Failing intermittently with: java.io.EOFException: Locator at HostAndPort[d5676dfc4e21:24193] did not respond. This is normal if the locator was shutdown. If it wasn't check its log for exceptions. at org.apache.geode.distributed.internal.tcpserver.TcpClient.requestToServer(TcpClient.java:202) at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.findCoordinator(GMSJoinLeave.java:1145) at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:331) at org.apache.geode.distributed.internal.membership.gms.GMSMembership.join(GMSMembership.java:573) at org.apache.geode.distributed.internal.membership.gms.GMSMembership.access$1300(GMSMembership.java:71) at org.apache.geode.distributed.internal.membership.gms.GMSMembership$ManagerImpl.joinDistributedSystem(GMSMembership.java:1972) at org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:242) at org.apache.geode.distributed.internal.membership.gms.GMSMembership.start(GMSMembership.java:1851) at org.apache.geode.distributed.internal.DistributionImpl.start(DistributionImpl.java:171) at org.apache.geode.distributed.internal.DistributionImpl.createDistribution(DistributionImpl.java:222) at org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:464) at org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:497) at org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:326) at org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:779) at org.apache.geode.distributed.internal.InternalDistributedSystem.access$200(InternalDistributedSystem.java:135) at org.apache.geode.distributed.internal.InternalDistributedSystem$Builder.build(InternalDistributedSystem.java:3033) at org.apache.geode.distributed.internal.InternalDistributedSystem.connectInternal(InternalDistributedSystem.java:290) at org.apache.geode.distributed.internal.InternalLocator.startDistributedSystem(InternalLocator.java:743) at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:388) at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:338) at org.apache.geode.distributed.Locator.startLocator(Locator.java:252) at org.apache.geode.distributed.Locator.startLocatorAndDS(Locator.java:139) at org.apache.geode.distributed.LocatorDUnitTest.startLocatorWithPortAndProperties(LocatorDUnitTest.java:1507) at org.apache.geode.distributed.LocatorDUnitTest.lambda$startVerifyAndStopLocator$e21733f2$1(LocatorDUnitTest.java:1539) at org.apache.geode.test.dunit.internal.IdentifiableRunnable.run(IdentifiableRunnable.java:41) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.geode.test.dunit.internal.MethodInvoker.executeObject(MethodInvoker.java:123) at org.apache.geode.test.dunit.internal.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:78) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:267) at org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2493) at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2864) at org.apache.geode.internal.InternalDataSerializer$1.readObject(InternalDataSerializer.java:301) at org.apache.geode.distributed.internal.tcpserver.TcpClient.requestToServer(TcpClient.java:197) ... 46 more

            People

              Unassigned Unassigned
              ukohlmeyer Udo Kohlmeyer
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: