[GEODE-8617] org.apache.geode.test.dunit.VM.bounce() fails to restart VM with different Geode version due to EOFException

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.9.0, 1.11.0, 1.12.0
• Assignee: Unassigned
• Reporter: Donal Evans
• Created: 15/Oct/20 18:47
• Updated: 05/Apr/22 18:25

Description

This failure shows up in RollingUpgrade and BackwardsCompatibility tests occasionally when attempting to bounce a VM and restart it with a different version of Geode. The initial failure shows up as an EOFException, but subsequent runs of the same test with other versions (as part of the RollingUpgrade/BackwardsCompatibility test) fail with ConnectException: Connection refused (Connection refused).

Two other tickets (GEODE-7142 and GEODE-6337) exist describing this failure in specific RollingUpgrade tests, but as it seems to be more general and not related to a specific test, this ticket has been created to consolidate them.

An example failure from a BackwardsCompatibility test is attached below.

> Task :geode-core:upgradeTest

org.apache.geode.internal.cache.TxCommitMessageBCOldClientToServerTxPartitionTest > test[9] FAILED
    org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.VM$$Lambda$30/1525819532.run in VM 3 running on Host fad43bff7558 with 5 VMs with version 1.9.1

        Caused by:
        java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is: 
          java.io.EOFException

            Caused by:
            java.io.EOFException

    org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.internal.cache.TxCommitMessageBCTestBase$$Lambda$44/1148702340.run in VM 3 running on Host fad43bff7558 with 5 VMs with version 1.9.1

        Caused by:
        java.lang.IllegalStateException: VM not available: VM 3 running on Host fad43bff7558 with 5 VMs with version 1.9.1

org.apache.geode.internal.cache.TxCommitMessageBCOldClientToServerTxPartitionTest > test[10] FAILED
    org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.VM$$Lambda$29/478086753.call in VM 3 running on Host fad43bff7558 with 5 VMs with version 1.9.1

        Caused by:
        java.rmi.ConnectException: Connection refused to host: 172.17.0.49; nested exception is: 
          java.net.ConnectException: Connection refused (Connection refused)

            Caused by:
            java.net.ConnectException: Connection refused (Connection refused)

    org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.internal.cache.TxCommitMessageBCTestBase$$Lambda$44/1148702340.run in VM 3 running on Host fad43bff7558 with 5 VMs with version 1.9.1

        Caused by:
        java.rmi.ConnectException: Connection refused to host: 172.17.0.49; nested exception is: 
          java.net.ConnectException: Connection refused (Connection refused)

            Caused by:
            java.net.ConnectException: Connection refused (Connection refused)

org.apache.geode.internal.cache.TxCommitMessageBCOldClientToServerTxPartitionTest > test[11] FAILED
    org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.internal.IdentifiableRunnable.run in VM 3 running on Host fad43bff7558 with 5 VMs with version 1.9.1

        Caused by:
        java.rmi.ConnectException: Connection refused to host: 172.17.0.49; nested exception is: 
          java.net.ConnectException: Connection refused (Connection refused)

            Caused by:
            java.net.ConnectException: Connection refused (Connection refused)

    org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.internal.cache.TxCommitMessageBCTestBase$$Lambda$44/1148702340.run in VM 3 running on Host fad43bff7558 with 5 VMs with version 1.9.1

        Caused by:
        java.rmi.ConnectException: Connection refused to host: 172.17.0.49; nested exception is: 
          java.net.ConnectException: Connection refused (Connection refused)

            Caused by:
            java.net.ConnectException: Connection refused (Connection refused)

org.apache.geode.internal.cache.TxCommitMessageBCOldClientToServerTxPartitionTest > test[12] FAILED
    org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.internal.IdentifiableRunnable.run in VM 3 running on Host fad43bff7558 with 5 VMs with version 1.9.1

        Caused by:
        java.rmi.ConnectException: Connection refused to host: 172.17.0.49; nested exception is: 
          java.net.ConnectException: Connection refused (Connection refused)

            Caused by:
            java.net.ConnectException: Connection refused (Connection refused)

    org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.internal.cache.TxCommitMessageBCTestBase$$Lambda$44/1148702340.run in VM 3 running on Host fad43bff7558 with 5 VMs with version 1.9.1

        Caused by:
        java.rmi.ConnectException: Connection refused to host: 172.17.0.49; nested exception is: 
          java.net.ConnectException: Connection refused (Connection refused)

            Caused by:
            java.net.ConnectException: Connection refused (Connection refused)

416 tests completed, 4 failed, 26 skipped

=============== Test Results URI =================
http://files.apachegeode-ci.info/builds/apache-support-1-12-main/1.12.1-build.0108/test-results/upgradeTest/1601570081/
=========================================

Test report artifacts from this job are available at:

http://files.apachegeode-ci.info/builds/apache-support-1-12-main/1.12.1-build.0108/test-artifacts/1601570081/upgradetestfiles-OpenJDK8-1.12.1-build.0108.tgz

Issue Links

• relates to: GEODE-6337 Rolling upgrade test fails on JDK11 in CI (sometimes)
• relates to: GEODE-7142 CI failure: RollingUpgradeRollSingleLocatorWithMultipleServersReplicatedRegion > testRollSingleLocatorWithMultipleServersReplicatedRegion[from_v160] FAILED
• relates to: GEODE-8642 CI Failure: RollingUpgradeQueryReturnsCorrectResultsAfterServersRollOverOnPersistentPartitionRegion.luceneQueryReturnsCorrectResultsAfterServersRollOverOnPersistentPartitionRegion