[CASSANDRA-15667] StreamResultFuture check for completeness is inconsistent, leading to races

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.21, 3.11.7, 4.0-beta1, 4.0
• Component/s: Legacy/Streaming and Messaging
• Bug Category: Correctness
• Severity: Normal
• Complexity: Normal
• Discovered By: Adhoc Test
• Platform: All
• Since Version: 3.0.0

Source Control Link

https://github.com/apache/cassandra/commit/a4b6deb906d846c2532174fcebfce0398b4b3c29

Test and Documentation Plan

https://app.circleci.com/pipelines/github/maxtomassi/cassandra?branch=15667-4.0

It seems like JVM dtests fail to run properly. Lots of logs like this:

[junit-timeout] Testcase: prepareRPCTimeout[PARALLEL/true](org.apache.cassandra.distributed.test.PreviewRepairCoordinatorTimeoutTest):	Caused an ERROR
[junit-timeout] org.apache.cassandra.distributed.api.NodeToolResult$Asserts.errorContains([Ljava/lang/String;)Lorg/apache/cassandra/distributed/api/NodeToolResult$Asserts;
[junit-timeout] java.lang.NoSuchMethodError: org.apache.cassandra.distributed.api.NodeToolResult$Asserts.errorContains([Ljava/lang/String;)Lorg/apache/cassandra/distributed/api/NodeToolResult$Asserts;
[junit-timeout] 	at org.apache.cassandra.distributed.test.RepairCoordinatorTimeout.lambda$prepareRPCTimeout$0(RepairCoordinatorTimeout.java:45)
[junit-timeout] 	at org.apache.cassandra.utils.AssertUtil.lambda$assertTimeoutPreemptively$0(AssertUtil.java:39)
[junit-timeout] 	at org.apache.cassandra.utils.AssertUtil.lambda$assertTimeoutPreemptively$1(AssertUtil.java:67)
[junit-timeout] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit-timeout] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit-timeout] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit-timeout] 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[junit-timeout] 	at java.lang.Thread.run(Thread.java:748)

• Assignee: Massimiliano Tomassi
• Reporter: Sergio Bossa

Authors

Massimiliano Tomassi

Reviewers

Sergio Bossa, Zhao Yang

• Created: 27/Mar/20 11:23
• Updated: 21/Dec/20 08:03
• Resolved: 25/May/20 14:54

Description

StreamResultFuture#maybeComplete() uses StreamCoordinator#hasActiveSessions() to determine if all sessions are completed, but then accesses each session state via StreamCoordinator#getAllSessionInfo(): this is inconsistent, as the former relies on the actual StreamSession state, while the latter on the SessionInfo state, and the two are concurrently updated with no coordination whatsoever.

This leads to races, i.e. apparent in some dtest spurious failures, such as TestBootstrap.resumable_bootstrap_test in CASSANDRA-15614 cc e.dimitrova.

Issue Links

• relates to: CASSANDRA-15952 TestBootstrap::test_bootstrap_binary_disabled_resumable_bootstrap checks non-existent log messages for 3.0 and 3.11