[CASSANDRA-18218] OOM while running ShortAccordSimulationTest

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: NA
• Component/s: Accord, Test/burn
• Labels: simulator
• Epic Link: CEP-15: Accord Beta
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: Fuzz Test
• Platform: All
• Since Version: NA

Source Control Link

https://github.com/apache/cassandra-accord/commit/0cc9e273b2eaa37d82a1ae1ac2681aec65aa0f6d

Test and Documentation Plan

n/a (test fix)

• Assignee: Caleb Rackliffe
• Reporter: Caleb Rackliffe

Authors

Benedict Elliott Smith

Reviewers

David Capwell

• Created: 01/Feb/23 20:43
• Updated: 02/Feb/23 21:18
• Resolved: 02/Feb/23 21:18

Description

ShortAccordSimulationTest seems to consistently run out of heap when run locally. This is true even when we scale the number of threads and duration down...

AccordSimulationRunner.main(new String[] { "run", "-n", "3", "-t", "10", "--cluster-action-limit", "-1", "-c", "2", "-s", "10"});

ERROR [CommandStore[0]:1] node1 CS:[0] OP:0x8fb4f8f1 2023-01-31 17:16:27,143 Operation AsyncOperation{RUNNING}-0x8fb4f8f1 failed
java.lang.OutOfMemoryError: Java heap space
   at java.util.Arrays.copyOf(Arrays.java:3181)
   at accord.local.Command$NotifyWaitingOn.push(Command.java:794)
   at accord.local.Command$NotifyWaitingOn.accept(Command.java:757)
   at accord.local.Command.maybeExecute(Command.java:600)
   at accord.local.Command.onChange(Command.java:546)
   at org.apache.cassandra.service.accord.ListenerProxy$CommandListenerProxy.lambda$onChange$0(ListenerProxy.java:148)
   at org.apache.cassandra.service.accord.ListenerProxy$CommandListenerProxy$$Lambda$4606/1457728285.accept(Unknown Source)
   at org.apache.cassandra.service.accord.async.AsyncOperation$ForConsumer.apply(AsyncOperation.java:261)
   at org.apache.cassandra.service.accord.async.AsyncOperation$ForConsumer.apply(AsyncOperation.java:248)
   at org.apache.cassandra.service.accord.async.AsyncOperation.runInternal(AsyncOperation.java:154)
   at org.apache.cassandra.service.accord.async.AsyncOperation.run(AsyncOperation.java:194)
   at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
   at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68)
   at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:584)
   at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus$$Lambda$768/827906088.run(Unknown Source)
   at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   at java.lang.Thread.run(Thread.java:748)

JVM args make it seem like we’re passing both a 1 GiB and 8 GiB heap size, although that doesn’t seem to have any bearing on the result. Setting only 8 GiB just takes longer to hit the same problem.

INFO  [isolatedExecutor:2] node1 2023-01-31 17:15:56,940 JVM Arguments: [-Dstorage-config=/Users/maedhroz/Forks/cassandra/test/conf, -Djava.awt.headless=true, -javaagent:/Users/maedhroz/Forks/cassandra/lib/jamm-0.3.2.jar, -ea, -Djava.io.tmpdir=/var/folders/4d/zfjs7m7s6x5_l93k33r5k6680000gn/T/, -Dcassandra.debugrefcount=true, -Xss384k, -XX:SoftRefLRUPolicyMSPerMB=0, -XX:HeapDumpPath=build/test, -Dcassandra.test.driver.connection_timeout_ms=10000, -Dcassandra.test.driver.read_timeout_ms=24000, -Dcassandra.memtable_row_overhead_computation_step=100, -Dcassandra.test.use_prepared=true, -Dcassandra.test.sstableformatdevelopment=true, -Djava.security.egd=file:/dev/urandom, -Dcassandra.testtag=, -Dcassandra.keepBriefBrief=${cassandra.keepBriefBrief}, -Dcassandra.strict.runtime.checks=true, -Dcassandra.reads.thresholds.coordinator.defensive_checks_enabled=true, -DQT_SHRINKS=0, -Dlogback.configurationFile=test/conf/logback-simulator.xml, -Dcassandra.ring_delay_ms=10000, -Dcassandra.tolerate_sstable_size=true, -Dcassandra.skip_sync=true, -Dcassandra.debugrefcount=false, -Dcassandra.test.simulator.determinismcheck=strict, -Dcassandra.test.simulator.print_asm=none, -javaagent:/Users/maedhroz/Forks/cassandra/build/test/lib/jars/simulator-asm.jar, -Xbootclasspath/a:/Users/maedhroz/Forks/cassandra/build/test/lib/jars/simulator-bootstrap.jar, -XX:ActiveProcessorCount=4, -XX:-TieredCompilation, -XX:-BackgroundCompilation, -XX:CICompilerCount=1, -XX:Tier4CompileThreshold=1000, -XX:ReservedCodeCacheSize=256M, -Xmx8G, -Xmx1024m]

Issue Links

• blocks: CASSANDRA-18196 Initial Merge of Accord Feature Branch to Trunk