Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-16002

jvm upgrade dtests fail on java 11 caused by bad initialization order of DatabaseDescriptor and FileUtils

    XMLWordPrintableJSON

    Details

      Description

      In FileUtils we check to see if we have access to some classes (specifically to set org.apache.cassandra.io.util.FileUtils#isCleanerAvailable), which can fail in java 11. This is fine with CassandraDaemon as it will just log the failure, but in in-jvm dtests this can fail to startup an instance with the following

      java.lang.RuntimeException: java.lang.RuntimeException: java.lang.AssertionError: network topology must be assigned before using snitch
      	at org.apache.cassandra.distributed.impl.IsolatedExecutor.waitOn(IsolatedExecutor.java:209)
      	at org.apache.cassandra.distributed.impl.IsolatedExecutor.lambda$sync$7(IsolatedExecutor.java:112)
      	at org.apache.cassandra.distributed.impl.Instance.startup(Instance.java:592)
      	at org.apache.cassandra.distributed.impl.AbstractCluster$Wrapper.startup(AbstractCluster.java:209)
      	at org.apache.cassandra.distributed.impl.AbstractCluster$Wrapper.startup(AbstractCluster.java:200)
      	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:179)
      	at org.apache.cassandra.distributed.upgrade.UpgradeTest.upgradeTest(UpgradeTest.java:50)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      Caused by: java.lang.RuntimeException: java.lang.AssertionError: network topology must be assigned before using snitch
      	at org.apache.cassandra.distributed.impl.Instance.lambda$startup$7(Instance.java:590)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:83)
      	at java.base/java.lang.Thread.run(Thread.java:834)
      Caused by: java.lang.AssertionError: network topology must be assigned before using snitch
      	at org.apache.cassandra.distributed.impl.DistributedTestSnitch.getDatacenter(DistributedTestSnitch.java:90)
      	at org.apache.cassandra.distributed.impl.DistributedTestSnitch.getDatacenter(DistributedTestSnitch.java:85)
      	at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:118)
      	at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:488)
      	at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:137)
      	at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:102)
      	at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:60)
      	at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:78)
      	at org.apache.cassandra.distributed.impl.Instance.lambda$startup$7(Instance.java:509)
      

      The exception isn’t clear, but what is happening is the following

      static
      {
          boolean canClean = false;
          try
          {
              ByteBuffer buf = ByteBuffer.allocateDirect(1);
              ((DirectBuffer) buf).cleaner().clean();
              canClean = true;
          }
          catch (Throwable t)
          {
              JVMStabilityInspector.inspectThrowable(t);
              logger.info("Cannot initialize un-mmaper.  (Are you using a non-Oracle JVM?)  Compacted data files will not be removed promptly.  Consider using an Oracle JVM or using standard disk access mode");
          }
          canCleanDirectBuffers = canClean;
      }
      

      JVMStabilityInspector will check the throwable which will eventually call org.apache.cassandra.config.DatabaseDescriptor#getDiskFailurePolicy which will try to load the configs and fail

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dcapwell David Capwell
                Reporter:
                dcapwell David Capwell
                Authors:
                David Capwell
                Reviewers:
                Jon Meredith, Jordan West
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: