[CASSANDRA-16002] jvm upgrade dtests fail on java 11 caused by bad initialization order of DatabaseDescriptor and FileUtils

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.22, 3.11.8, 4.0-beta2, 4.0
• Component/s: Test/dtest/java
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Low Hanging Fruit
• Discovered By: Unit Test
• Platform: All
• Since Version: 3.0.21

Source Control Link

https://github.com/apache/cassandra/commit/624d01660bdad4dc924717f4c602ce6241c0c825

Test and Documentation Plan

manual testing

• Assignee: David Capwell
• Reporter: David Capwell

Authors

David Capwell

Reviewers

Jon Meredith, Jordan West

• Created: 30/Jul/20 18:28
• Updated: 03/Jan/21 17:03
• Resolved: 31/Jul/20 02:31

Description

In FileUtils we check to see if we have access to some classes (specifically to set org.apache.cassandra.io.util.FileUtils#isCleanerAvailable), which can fail in java 11. This is fine with CassandraDaemon as it will just log the failure, but in in-jvm dtests this can fail to startup an instance with the following

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.AssertionError: network topology must be assigned before using snitch
	at org.apache.cassandra.distributed.impl.IsolatedExecutor.waitOn(IsolatedExecutor.java:209)
	at org.apache.cassandra.distributed.impl.IsolatedExecutor.lambda$sync$7(IsolatedExecutor.java:112)
	at org.apache.cassandra.distributed.impl.Instance.startup(Instance.java:592)
	at org.apache.cassandra.distributed.impl.AbstractCluster$Wrapper.startup(AbstractCluster.java:209)
	at org.apache.cassandra.distributed.impl.AbstractCluster$Wrapper.startup(AbstractCluster.java:200)
	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:179)
	at org.apache.cassandra.distributed.upgrade.UpgradeTest.upgradeTest(UpgradeTest.java:50)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Caused by: java.lang.RuntimeException: java.lang.AssertionError: network topology must be assigned before using snitch
	at org.apache.cassandra.distributed.impl.Instance.lambda$startup$7(Instance.java:590)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:83)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.AssertionError: network topology must be assigned before using snitch
	at org.apache.cassandra.distributed.impl.DistributedTestSnitch.getDatacenter(DistributedTestSnitch.java:90)
	at org.apache.cassandra.distributed.impl.DistributedTestSnitch.getDatacenter(DistributedTestSnitch.java:85)
	at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:118)
	at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:488)
	at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:137)
	at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:102)
	at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:60)
	at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:78)
	at org.apache.cassandra.distributed.impl.Instance.lambda$startup$7(Instance.java:509)

The exception isn’t clear, but what is happening is the following

static
{
    boolean canClean = false;
    try
    {
        ByteBuffer buf = ByteBuffer.allocateDirect(1);
        ((DirectBuffer) buf).cleaner().clean();
        canClean = true;
    }
    catch (Throwable t)
    {
        JVMStabilityInspector.inspectThrowable(t);
        logger.info("Cannot initialize un-mmaper.  (Are you using a non-Oracle JVM?)  Compacted data files will not be removed promptly.  Consider using an Oracle JVM or using standard disk access mode");
    }
    canCleanDirectBuffers = canClean;
}

JVMStabilityInspector will check the throwable which will eventually call org.apache.cassandra.config.DatabaseDescriptor#getDiskFailurePolicy which will try to load the configs and fail

Issue Links

• relates to: CASSANDRA-15981 jvm-dtests crash on java 11
• links to: GitHub Branch: 3.0
• links to: GitHub Branch: 3.11
• links to: GitHub Branch: trunk