Details
-
Bug
-
Status: Reopened
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
Description
This test fails due to something not cleaning itself properly. Undetermined what the problem is, but it will run perfectly by itself everytime, but once run inside of the TestClass it fails
Attachments
Issue Links
- causes
-
GEODE-10066 SSL handshake failures on 1 locator prevents connection pool from trying other locators
- Closed
- relates to
-
GEODE-2412 Builds are failing in pipeline due to SSL locator tests failing.
- Closed
-
GEODE-420 locator ssl configuration
- Closed
- links to
Activity
This test failed during a CI run
org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedRunnable.run in VM 2 running on Host cc4-rh6.gemstone.com with 4 VMs at org.apache.geode.test.dunit.VM.invoke(VM.java:389) at org.apache.geode.test.dunit.VM.invoke(VM.java:355) at org.apache.geode.test.dunit.VM.invoke(VM.java:280) at org.apache.geode.distributed.LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL(LocatorDUnitTest.java:481) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54) at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.geode.SystemConnectException: Problem starting up membership services at org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:116) at org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:92) at org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1092) at org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1144) at org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:521) at org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:659) at org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299) at org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:237) at org.apache.geode.distributed.internal.InternalLocator.startDistributedSystem(InternalLocator.java:746) at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:395) at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:336) at org.apache.geode.distributed.Locator.startLocator(Locator.java:334) at org.apache.geode.distributed.Locator.startLocatorAndDS(Locator.java:166) at org.apache.geode.distributed.LocatorDUnitTest.startLocator(LocatorDUnitTest.java:627) at org.apache.geode.distributed.LocatorDUnitTest.lambda$testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL$1965b9f9$1(LocatorDUnitTest.java:481) at org.apache.geode.test.dunit.NamedRunnable.run(NamedRunnable.java:33) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at hydra.MethExecutor.executeObject(MethExecutor.java:268) at org.apache.geode.test.dunit.standalone.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:82) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$95(TCPTransport.java:683) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) ... 3 more Caused by: org.apache.geode.distributed.internal.tcpserver.LocatorCancelException at org.apache.geode.distributed.internal.tcpserver.TcpClient.getServerVersion(TcpClient.java:272) at org.apache.geode.distributed.internal.tcpserver.TcpClient.requestToServer(TcpClient.java:157) at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$TcpClientWrapper.sendCoordinatorFindRequest(GMSJoinLeave.java:1122) at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.findCoordinator(GMSJoinLeave.java:1037) at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:302) at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:662) at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:749) at org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:183) at org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:104) ... 36 more
this ticket describes the test as Flaky but the test has not been annotated as being flaky and so is still being run with regular unit tests
Sometimes, there is a RMIException thrown to the actual caller as a result of the connect failing, rather than it happening asynchronously. I haven't gone deep enough into the locator to find the reason for this, but a fix for the test is to catch the exception before it propagates as an RMIException.
Of note: the Locator makes two socket connections, one to get the header, and then one to get the rest of the connection. I feel that this could be better if we didn't make so many connections, but that would break backward compatibility.
GitHub user galen-pivotal opened a pull request:
https://github.com/apache/geode/pull/360
GEODE-1793 Fix a flaky test, clean up SSL tests.
- Use less code.
- Get random port numbers a bit more robustly.
- Check to make sure that locator2 has died at the end.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/galen-pivotal/incubator-geode feature/GEODE-1793
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/geode/pull/360.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #360
commit b5a9f347b73cdbc0a5e401db5e7b11447a062045
Author: Galen O'Sullivan <gosullivan@pivotal.io>
Date: 2017-01-11T03:56:42Z
GEODE-1793 Fix a flaky test, clean up SSL tests.
- Use less code.
- Get random port numbers a bit more robustly.
- Check to make sure that locator2 has died at the end.
Commit 8426c675c464896eab659a86fdb7a8c17706c607 in geode's branch refs/heads/develop from gosullivan
[ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=8426c67 ]
GEODE-1793: Fix a flaky test, clean up SSL tests. This closes #360
- Add a couple of common test methods.
- Get random port numbers a bit more robustly.
- Check to make sure that locator2 has died at the end.
I managed to get this test to fail & see that the second locator, in vm_2, was supposed to shut down but instead it created its own cluster:
[vm2] [info 2017/03/02 15:19:56.969 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Starting membership services [vm2] [info 2017/03/02 15:19:56.979 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] JGroups channel created (took 9ms) [vm2] [info 2017/03/02 15:19:56.980 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] GemFire P2P Listener started on /10.118.32.92:39824 [vm2] [info 2017/03/02 15:19:56.980 PST <Geode Failure Detection Server thread 0> tid=0x21c] Started failure detection server thread on /10.118.32.92:59150. [vm2] [info 2017/03/02 15:19:56.980 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Peer locator is connecting to local membership services with ID trout(2830:locator)<ec>:32771 [vm2] [info 2017/03/02 15:19:57.164 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] This member is becoming the membership coordinator with address trout(2830:locator)<ec>:32771
It was correctly configured with two locator addresses and its SSL configuration appeared to be okay. The other locator, in vm_1, correctly had SSL disabled. The SSL locator was unable to recover from the non-SSL locator:
[vm2] [info 2017/03/02 15:19:56.961 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] GemFire peer location service starting. Other locators: trout.gemstone.com[46843] Locators preferred as coordinators: false Network partition detection enabled: false View persistence file: locator0view.dat [vm2] [info 2017/03/02 15:19:56.961 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Peer locator attempting to recover from trout.gemstone.com/10.118.32.92:46843 [vm2] [info 2017/03/02 15:19:56.963 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Peer locator was unable to recover state from this locator [vm2] [info 2017/03/02 15:19:56.963 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] recovery file not found: /export/trout1/users/bschuchardt/devel/gfdev/open/geode-core/build/distributedTest/dunit/vm2/locator0view.dat [vm2] [info 2017/03/02 15:19:56.963 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Starting distributed system
So this appears to be a problem with Geode, not the test. It is the Geode functionality that is "flaky", working most of the time but not 100%.
In another run with modified logging I can see that the SSL locator did not see an SSLHandshakeException as expected but merely an IOException
[vm2] [info 2017/03/02 15:40:00.701 PST <RMI TCP Connection(1)-10.118.32.92> tid=0x13] Peer locator could not recover membership view from trout.gemstone.com/10.118.32.92:38254: Connection reset
So this appears to be an inconsistency in SSL implementations used by the JVM.
Commit 866dc5ca1583c5fab49ec96c48d261c0367427f3 in geode's branch refs/heads/feature/GEODE-1793 from bschuchardt
[ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=866dc5c ]
GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL
This was a product issue. When the locator using plain-text sockets is
contacted by a TcpClient using SSL the locator often just closes the socket.
On some platforms this causes a SSLHandshakeException but on others it
just causes a "SocketException: connection reset". Writing some text to
the socket forces the TcpClient to get a SSLException (which is the superclass
of SSLHandshakeException).
The test class is still marked as Flaky due to GEODE-2542.
GitHub user bschuchardt opened a pull request:
https://github.com/apache/geode/pull/412
GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOther…
This was a product issue. When the locator using plain-text sockets is
contacted by a TcpClient using SSL the locator often just closes the socket.
On some platforms this causes a SSLHandshakeException but on others it
just causes a "SocketException: connection reset". Writing some text to
the socket forces the TcpClient to get a SSLException (which is the superclass
of SSLHandshakeException).
The test class is still marked as Flaky due to GEODE-2542.
I deleted one of the tests in LocatorDUnitTest as it wasn't doing any useful validation and really served no purpose.
I also increased the joinTimeout in this test. The original 1-second timeout was intended to make the tests run faster but I think it's probably the source of some of the flaky-ness in this set of tests. Some of them were also overriding the joinTimeout established by the DUnit framework, so that was actually a bad thing to be doing. The tests all run in a few seconds with the default joinTimeout setting anyway.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/geode feature/GEODE-1793
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/geode/pull/412.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #412
commit 866dc5ca1583c5fab49ec96c48d261c0367427f3
Author: Bruce Schuchardt <bschuchardt@pivotal.io>
Date: 2017-03-03T21:47:42Z
GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL
This was a product issue. When the locator using plain-text sockets is
contacted by a TcpClient using SSL the locator often just closes the socket.
On some platforms this causes a SSLHandshakeException but on others it
just causes a "SocketException: connection reset". Writing some text to
the socket forces the TcpClient to get a SSLException (which is the superclass
of SSLHandshakeException).
The test class is still marked as Flaky due to GEODE-2542.
Github user bschuchardt commented on the issue:
https://github.com/apache/geode/pull/412
There are "spotless" problems I'm cleaning up, and I'm removing the commented out code from the test.
Github user galen-pivotal commented on a diff in the pull request:
https://github.com/apache/geode/pull/412#discussion_r104258182
— Diff: geode-core/src/main/java/org/apache/geode/distributed/internal/tcpserver/TcpServer.java —
@@ -360,6 +360,13 @@ private void processRequest(final Socket sock)
else {
// Close the socket. We can not accept requests from a newer version
+ try {
+ sock.getOutputStream().write("unknown protocol version".getBytes());
— End diff –
Is there any risk of this being interpreted by anything other than garbage on the other side?
Github user galen-pivotal commented on a diff in the pull request:
https://github.com/apache/geode/pull/412#discussion_r104258974
— Diff: geode-core/src/main/java/org/apache/geode/distributed/internal/tcpserver/TcpServer.java —
@@ -77,9 +77,9 @@
- <p>
- This should be incremented if the gossip message structures change
- <p>
- * 1000 - gemfire 5.5 - using java serialization 1001 - 5.7 - using DataSerializable and
-
- End diff –
-
Did spotless break this? 😮
Github user bschuchardt commented on a diff in the pull request:
https://github.com/apache/geode/pull/412#discussion_r104259153
— Diff: geode-core/src/main/java/org/apache/geode/distributed/internal/tcpserver/TcpServer.java —
@@ -77,9 +77,9 @@
- <p>
- This should be incremented if the gossip message structures change
- <p>
- * 1000 - gemfire 5.5 - using java serialization 1001 - 5.7 - using DataSerializable and
-
- End diff –
-
I didn't check but that's my assumption. It likes the comment with the HTML breaks in place.
Github user bschuchardt commented on a diff in the pull request:
https://github.com/apache/geode/pull/412#discussion_r104260160
— Diff: geode-core/src/main/java/org/apache/geode/distributed/internal/tcpserver/TcpServer.java —
@@ -360,6 +360,13 @@ private void processRequest(final Socket sock)
else {
// Close the socket. We can not accept requests from a newer version
+ try {
+ sock.getOutputStream().write("unknown protocol version".getBytes());
— End diff –
This situation should never happen. When we fetch the protocol version of a Locator we use v5.7. Then we use the locator's protocol version in future communications. We will never send a higher version number than the locator knows how to handle.
For SSL, this text is nothing like the response to a HelloClient packet. On machines I've tested on it results in either a SSLHandshakeException or a SSLException hinting that the other end might be using plain text sockets.
Github user bschuchardt commented on the issue:
https://github.com/apache/geode/pull/412
remote: geode git commit: GEODE_1793 spotless fixes and removal of dead code
remote: geode git commit: GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL
To https://git-wip-us.apache.org/repos/asf/geode.git
c09a856..4112204 develop -> develop
Commit 064362e90e150ad4ef5b269fab78f0cf2d6e5f4f in geode's branch refs/heads/develop from bschuchardt
[ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=064362e ]
GEODE-1793 LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL
This was a product issue. When the locator using plain-text sockets is
contacted by a TcpClient using SSL the locator often just closes the socket.
On some platforms this causes a SSLHandshakeException but on others it
just causes a "SocketException: connection reset". Writing some text to
the socket forces the TcpClient to get a SSLException (which is the superclass
of SSLHandshakeException).
The test class is still marked as Flaky due to GEODE-2542.
Failing intermittently with:
java.io.EOFException: Locator at HostAndPort[d5676dfc4e21:24193] did not respond. This is normal if the locator was shutdown. If it wasn't check its log for exceptions. at org.apache.geode.distributed.internal.tcpserver.TcpClient.requestToServer(TcpClient.java:202) at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.findCoordinator(GMSJoinLeave.java:1145) at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:331) at org.apache.geode.distributed.internal.membership.gms.GMSMembership.join(GMSMembership.java:573) at org.apache.geode.distributed.internal.membership.gms.GMSMembership.access$1300(GMSMembership.java:71) at org.apache.geode.distributed.internal.membership.gms.GMSMembership$ManagerImpl.joinDistributedSystem(GMSMembership.java:1972) at org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:242) at org.apache.geode.distributed.internal.membership.gms.GMSMembership.start(GMSMembership.java:1851) at org.apache.geode.distributed.internal.DistributionImpl.start(DistributionImpl.java:171) at org.apache.geode.distributed.internal.DistributionImpl.createDistribution(DistributionImpl.java:222) at org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:464) at org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:497) at org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:326) at org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:779) at org.apache.geode.distributed.internal.InternalDistributedSystem.access$200(InternalDistributedSystem.java:135) at org.apache.geode.distributed.internal.InternalDistributedSystem$Builder.build(InternalDistributedSystem.java:3033) at org.apache.geode.distributed.internal.InternalDistributedSystem.connectInternal(InternalDistributedSystem.java:290) at org.apache.geode.distributed.internal.InternalLocator.startDistributedSystem(InternalLocator.java:743) at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:388) at org.apache.geode.distributed.internal.InternalLocator.startLocator(InternalLocator.java:338) at org.apache.geode.distributed.Locator.startLocator(Locator.java:252) at org.apache.geode.distributed.Locator.startLocatorAndDS(Locator.java:139) at org.apache.geode.distributed.LocatorDUnitTest.startLocatorWithPortAndProperties(LocatorDUnitTest.java:1507) at org.apache.geode.distributed.LocatorDUnitTest.lambda$startVerifyAndStopLocator$e21733f2$1(LocatorDUnitTest.java:1539) at org.apache.geode.test.dunit.internal.IdentifiableRunnable.run(IdentifiableRunnable.java:41) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.geode.test.dunit.internal.MethodInvoker.executeObject(MethodInvoker.java:123) at org.apache.geode.test.dunit.internal.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:78) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:267) at org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2493) at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2864) at org.apache.geode.internal.InternalDataSerializer$1.readObject(InternalDataSerializer.java:301) at org.apache.geode.distributed.internal.tcpserver.TcpClient.requestToServer(TcpClient.java:197) ... 46 more
fixed in dunit cleanup method