[GEODE-6337] Rolling upgrade test fails on JDK11 in CI (sometimes)

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.9.0, 1.13.6, 1.13.7
• Component/s: tests
• Labels: flaky jdk11
• Assignee: Unassigned
• Reporter: Bill Burcham
• Created: 30/Jan/19 00:12
• Updated: 04/Jan/22 19:10

Description

In this build:

https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/UpgradeTestOpenJDK11/builds/333

RollingUpgradeRollLocatorsWithOldServer.testRollLocatorsWithOldServer() fails, apparently trying to start up Geode version 1.3.0.

Have a look at the first two errors in the test output:

org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeRollLocatorsWithOldServer > testRollLocatorsWithOldServer[from_v130] FAILED
    org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.VM$$Lambda$157/0x0000000840246c40.run in VM 1 running on Host f634ce7e984c with 4 VMs
        at org.apache.geode.test.dunit.VM.executeMethodOnObject(VM.java:537)
        at org.apache.geode.test.dunit.VM.bounce(VM.java:497)
        at org.apache.geode.test.dunit.VM.bounce(VM.java:476)
        at org.apache.geode.test.dunit.internal.DUnitHost.getVM(DUnitHost.java:85)
        at org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeRollLocatorsWithOldServer.testRollLocatorsWithOldServer(RollingUpgradeRollLocatorsWithOldServer.java:39)

        Caused by:
        java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is: 
        	java.io.EOFException
            at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:236)
            at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
            at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:209)
            at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:161)
            at com.sun.proxy.$Proxy40.executeMethodOnObject(Unknown Source)
            at org.apache.geode.test.dunit.VM.executeMethodOnObject(VM.java:530)
            ... 4 more

            Caused by:
            java.io.EOFException
                at java.io.DataInputStream.readByte(DataInputStream.java:272)
                at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222)
                ... 9 more

    org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedRunnable.run in VM 1 running on Host f634ce7e984c with 4 VMs

        Caused by:
        java.lang.IllegalStateException: VM not available: VM 1 running on Host f634ce7e984c with 4 VMs

org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeRollLocatorsWithOldServer > testRollLocatorsWithOldServer[from_v150] FAILED
    org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.VM$$Lambda$156/0x0000000840247040.call in VM 1 running on Host f634ce7e984c with 4 VMs

        Caused by:
        java.rmi.ConnectException: Connection refused to host: 172.17.0.4; nested exception is: 
        	java.net.ConnectException: Connection refused (Connection refused)

            Caused by:
            java.net.ConnectException: Connection refused (Connection refused)

    org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.test.dunit.NamedRunnable.run in VM 1 running on Host f634ce7e984c with 4 VMs

        Caused by:
        java.rmi.ConnectException: Connection refused to host: 172.17.0.4; nested exception is: 
        	java.net.ConnectException: Connection refused (Connection refused)

            Caused by:
            java.net.ConnectException: Connection refused (Connection refused)

Notice that the first exception is caused by an EOF whereas the second one is "connection refused".

Dunno what version of Geode was running on that JVM when the attempt to bounce to 1.3.0 was made. But you can see in the test output that it wasn't one of these:

from_v130
from_v150
from_v160
from_v170
from_v180

We also noticed this in stderr. Dunno if it's important:

Geode version 140 is incompatible with Java 9 and higher.  Skipping this version.
[vm0_v100] WARNING: An illegal reflective access operation has occurred
[vm0_v100] WARNING: Illegal reflective access by org.apache.geode.internal.shared.NativeCalls (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.apache.geode/geode-core/1.0.0-incubating/28ed9510a84bfd48f1f01ac4bb6f2d5717fc01f/geode-core-1.0.0-incubating.jar) to field java.util.Collections$UnmodifiableMap.m
[vm0_v100] WARNING: Please consider reporting this to the maintainers of org.apache.geode.internal.shared.NativeCalls
[vm0_v100] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[vm0_v100] WARNING: All illegal access operations will be denied in a future release
[vm1_v100] WARNING: An illegal reflective access operation has occurred
[vm1_v100] WARNING: Illegal reflective access by org.apache.geode.internal.shared.NativeCalls (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.apache.geode/geode-core/1.0.0-incubating/28ed9510a84bfd48f1f01ac4bb6f2d5717fc01f/geode-core-1.0.0-incubating.jar) to field java.util.Collections$UnmodifiableMap.m
[vm1_v100] WARNING: Please consider reporting this to the maintainers of org.apache.geode.internal.shared.NativeCalls
[vm1_v100] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[vm1_v100] WARNING: All illegal access operations will be denied in a future release
[vm0_v110] WARNING: An illegal reflective access operation has occurred
[vm0_v110] WARNING: Illegal reflective access by org.apache.geode.internal.shared.NativeCalls (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.apache.geode/geode-core/1.1.0/5e920cbf22352f3f33b72eb12f1d560c058fafab/geode-core-1.1.0.jar) to field java.util.Collections$UnmodifiableMap.m
[vm0_v110] WARNING: Please consider reporting this to the maintainers of org.apache.geode.internal.shared.NativeCalls
[vm0_v110] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[vm0_v110] WARNING: All illegal access operations will be denied in a future release
[vm1_v110] WARNING: An illegal reflective access operation has occurred
[vm1_v110] WARNING: Illegal reflective access by org.apache.geode.internal.shared.NativeCalls (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.apache.geode/geode-core/1.1.0/5e920cbf22352f3f33b72eb12f1d560c058fafab/geode-core-1.1.0.jar) to field java.util.Collections$UnmodifiableMap.m
[vm1_v110] WARNING: Please consider reporting this to the maintainers of org.apache.geode.internal.shared.NativeCalls
[vm1_v110] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[vm1_v110] WARNING: All illegal access operations will be denied in a future release
[vm0_v111] WARNING: An illegal reflective access operation has occurred
[vm0_v111] WARNING: Illegal reflective access by org.apache.geode.internal.shared.NativeCalls (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.apache.geode/geode-core/1.1.1/28cbc629c60f9ad2cd97e46194854a0232c2ef50/geode-core-1.1.1.jar) to field java.util.Collections$UnmodifiableMap.m
[vm0_v111] WARNING: Please consider reporting this to the maintainers of org.apache.geode.internal.shared.NativeCalls
[vm0_v111] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[vm0_v111] WARNING: All illegal access operations will be denied in a future release
[vm1_v111] WARNING: An illegal reflective access operation has occurred
[vm1_v111] WARNING: Illegal reflective access by org.apache.geode.internal.shared.NativeCalls (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.apache.geode/geode-core/1.1.1/28cbc629c60f9ad2cd97e46194854a0232c2ef50/geode-core-1.1.1.jar) to field java.util.Collections$UnmodifiableMap.m
[vm1_v111] WARNING: Please consider reporting this to the maintainers of org.apache.geode.internal.shared.NativeCalls
[vm1_v111] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[vm1_v111] WARNING: All illegal access operations will be denied in a future release
[vm0_v120] WARNING: An illegal reflective access operation has occurred
[vm0_v120] WARNING: Illegal reflective access by org.apache.geode.internal.shared.NativeCalls (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.apache.geode/geode-core/1.2.0/d6cb1f1aa8db8c6212560bbeafba80e752f640a9/geode-core-1.2.0.jar) to field java.util.Collections$UnmodifiableMap.m
[vm0_v120] WARNING: Please consider reporting this to the maintainers of org.apache.geode.internal.shared.NativeCalls
[vm0_v120] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[vm0_v120] WARNING: All illegal access operations will be denied in a future release
[vm1_v120] WARNING: An illegal reflective access operation has occurred
[vm1_v120] WARNING: Illegal reflective access by org.apache.geode.internal.shared.NativeCalls (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.apache.geode/geode-core/1.2.0/d6cb1f1aa8db8c6212560bbeafba80e752f640a9/geode-core-1.2.0.jar) to field java.util.Collections$UnmodifiableMap.m
[vm1_v120] WARNING: Please consider reporting this to the maintainers of org.apache.geode.internal.shared.NativeCalls
[vm1_v120] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[vm1_v120] WARNING: All illegal access operations will be denied in a future release

Issue Links

• is related to: GEODE-8617 org.apache.geode.test.dunit.VM.bounce() fails to restart VM with different Geode version due to EOFException