[CASSANDRA-17566] Fix flaky test - org.apache.cassandra.distributed.test.repair.ForceRepairTest.force

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.1-beta1, 4.1
• Component/s: Test/dtest/java
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: DTest
• Platform: All
• Since Version: 4.1-alpha

Source Control Link

https://github.com/apache/cassandra/commit/f809b6753cbbd27deab40679b99d956c8193fcf8

Test and Documentation Plan

run CI

• Assignee: Brandon Williams
• Reporter: Brandon Williams

Authors

Brandon Williams

Reviewers

David Capwell

• Created: 20/Apr/22 15:38
• Updated: 05/Oct/22 22:22
• Resolved: 07/Jun/22 10:47

Description

Seen on jenkins here: https://ci-cassandra.apache.org/job/Cassandra-trunk/1083/testReport/org.apache.cassandra.distributed.test.repair/ForceRepairTest/force_2/

 

and circle here:

https://app.circleci.com/pipelines/github/driftx/cassandra/440/workflows/42f936c7-2ede-4fbf-957c-5fb4e461dd90/jobs/5160/tests#failed-test-1

junit.framework.AssertionFailedError: nodetool command [repair, distributed_test_keyspace, --force, --full] was not successful
stdout:
[2022-04-20 15:11:01,402] Starting repair command #2 (1701a090-c0bc-11ec-9898-07c796ce6a49), repairing keyspace distributed_test_keyspace with repair options (parallelism: parallel, primary range: false, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], previewKind: NONE, # of ranges: 3, pull repair: false, force repair: true, optimise streams: false, ignore unreplicated keyspaces: false, repairPaxos: true, paxosOnly: false)
[2022-04-20 15:11:11,406] Repair command #2 failed with error Did not get replies from all endpoints.
[2022-04-20 15:11:11,408] Repair command #2 finished with error

stderr:
error: Repair job has failed with the error message: Repair command #2 failed with error Did not get replies from all endpoints.. Check the logs on the repair participants for further details
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message: Repair command #2 failed with error Did not get replies from all endpoints.. Check the logs on the repair participants for further details
	at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:137)
	at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
	at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
	at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
	at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)