[CASSANDRA-16754] Flaky o.a.c.distributed.test.SchemaTest

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.25, 3.11.11, 4.0-rc2, 4.0
• Component/s: Test/dtest/java
• Bug Category: Correctness - Test Failure
• Severity: Low
• Complexity: Normal
• Discovered By: DTest
• Platform: All
• Since Version: 3.0.25

Source Control Link

https://github.com/apache/cassandra/commit/f329c63d86e8e123be360004fb61bf6c08f8b01a

Test and Documentation Plan

Flaky test

• Assignee: Andres de la Peña
• Reporter: Andres de la Peña

Authors

Andres de la Peña

Reviewers

Andres de la Peña, Berenguer Blasi

• Created: 22/Jun/21 11:38
• Updated: 01/Aug/21 13:32
• Resolved: 22/Jun/21 14:09

Description

The JVM dtest org.apache.cassandra.distributed.test.SchemaTest is flaky:
https://ci-cassandra.apache.org/job/Cassandra-4.0.0/34/testReport/junit/org.apache.cassandra.distributed.test/SchemaTest/readRepairWithCompaction_2/

Error Message
FSWriteError in /home/cassandra/cassandra/tmp/dtests5018452609443646984/node2/commitlog/CommitLog-7-1624286958980.log
Stacktrace
java.lang.RuntimeException: FSWriteError in /home/cassandra/cassandra/tmp/dtests5018452609443646984/node2/commitlog/CommitLog-7-1624286958980.log
	at org.apache.cassandra.distributed.impl.Instance.lambda$startup$10(Instance.java:582)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: FSWriteError in /home/cassandra/cassandra/tmp/dtests5018452609443646984/node2/commitlog/CommitLog-7-1624286958980.log
	at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:256)
	at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:273)
	at org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.handleReplayedSegment(AbstractCommitLogSegmentManager.java:349)
	at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:178)
	at org.apache.cassandra.distributed.impl.Instance.lambda$startup$10(Instance.java:508)
Caused by: java.nio.file.NoSuchFileException: /home/cassandra/cassandra/tmp/dtests5018452609443646984/node2/commitlog/CommitLog-7-1624286958980.log
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249)
	at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
	at java.base/java.nio.file.Files.delete(Files.java:1142)
	at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:250)

Although it doesn't fail frequently on CI it's quite easy to reproduce it locally.

I think the failure is caused by the two tests on the class not waiting for the future returned by IInstance#shutdown().

Issue Links

• links to: PR 4.0.0