Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-17674

Test failure: org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Normal
    • Resolution: Unresolved
    • 4.0.x, 4.x
    • Test/dtest/java
    • None
    • Correctness - Test Failure
    • Normal
    • Normal
    • User Report
    • All
    • None

    Description

      The Java upgrade dtest org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage is ~68% flaky on 4.0 and ~2% flaky on trunk, at least in CircleCI:

      The error for 4.0 is:

      java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
      	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
      	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
      	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:227)
      	at org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage(DropCompactStorageTest.java:49)
      Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
      	at org.apache.cassandra.distributed.impl.IsolatedExecutor$ThrowingRunnable.lambda$toRunnable$0(IsolatedExecutor.java:235)
      	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:83)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      	at org.apache.cassandra.distributed.impl.IsolatedExecutor$ThrowingRunnable.lambda$toRunnable$0(IsolatedExecutor.java:231)
      Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
      	at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:62)
      	at org.apache.cassandra.distributed.impl.Instance.lambda$shutdown$28(Instance.java:810)
      	at org.apache.cassandra.distributed.impl.IsolatedExecutor.lambda$null$8(IsolatedExecutor.java:114)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      Caused by: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
      	at org.apache.cassandra.utils.ExecutorUtils.awaitTerminationUntil(ExecutorUtils.java:107)
      	at org.apache.cassandra.utils.ExecutorUtils.awaitTermination(ExecutorUtils.java:96)
      	at org.apache.cassandra.utils.ExecutorUtils.shutdownNowAndWait(ExecutorUtils.java:139)
      	at org.apache.cassandra.concurrent.StageManager.shutdownAndWait(StageManager.java:128)
      	at org.apache.cassandra.distributed.impl.Instance.lambda$null$26(Instance.java:803)
      	at org.apache.cassandra.distributed.impl.Instance.lambda$parallelRun$32(Instance.java:968)
      

      Whereas the error for trunk is:

      org.apache.cassandra.distributed.shared.ShutdownException: Uncaught exceptions were thrown during test
      	at org.apache.cassandra.distributed.impl.AbstractCluster.checkAndResetUncaughtExceptions(AbstractCluster.java:1056)
      	at org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1042)
      	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:244)
      	at org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage(DropCompactStorageTest.java:49)
      	Suppressed: java.lang.NullPointerException
      		at org.apache.cassandra.gms.Gossiper.getHostId(Gossiper.java:992)
      		at org.apache.cassandra.gms.Gossiper.getHostId(Gossiper.java:987)
      		at org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2512)
      		at org.apache.cassandra.service.StorageService.onChange(StorageService.java:2086)
      		at org.apache.cassandra.gms.Gossiper.markAsShutdown(Gossiper.java:479)
      		at org.apache.cassandra.gms.GossipShutdownVerbHandler.doVerb(GossipShutdownVerbHandler.java:37)
      		at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:69)
      		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      		at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
      		at java.lang.Thread.run(Thread.java:748)
      

      I haven't managed to reproduce it on 4.1, although more iterations could do it.

      This was discovered during CASSANDRA-17213.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              adelapena Andres de la Peña
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: