[CASSANDRA-17489] Test Failures: org.apache.cassandra.metrics and a few others failing (utests_system_keyspace_directory on circleci)

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 4.1-beta1
• Component/s: CI
• Bug Category: Correctness - Test Failure
• Severity: Critical
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Since Version: 4.1-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/cdc0ce1fa33756f091785b58d4cc8f50fa9e7d0e

Test and Documentation Plan

circle ci

• Assignee: Blake Eggleston
• Reporter: Ekaterina Dimitrova

Authors

Blake Eggleston

Reviewers

Benjamin Lerer, Berenguer Blasi

• Created: 25/Mar/22 19:39
• Updated: 05/Oct/22 22:22
• Resolved: 12/Jul/22 12:18

Description

There are 10 tests failing in CircleCI - utests_system_keyspace_directory
This test target is not presented in Jenkins.
 
The tests fail for the same reason:

java.lang.RuntimeException: java.nio.file.NoSuchFileException: build/test/cassandra/data/system/_paxos_repair_state/ballot.meta.tmp at org.apache.cassandra.io.util.SequentialWriter.openChannel(SequentialWriter.java:128) at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:161) at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:150) at org.apache.cassandra.service.paxos.uncommitted.PaxosBallotTracker.flush(PaxosBallotTracker.java:130) at org.apache.cassandra.service.paxos.uncommitted.PaxosStateTracker.maybeRebuild(PaxosStateTracker.java:309) at org.apache.cassandra.service.paxos.PaxosState.maybeRebuildUncommittedState(PaxosState.java:122) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:368) at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:655) at org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:50) at org.apache.cassandra.metrics.CQLMetricsTest.setup(CQLMetricsTest.java:53) Caused by: java.nio.file.NoSuchFileException: build/test/cassandra/data/system/_paxos_repair_state/ballot.meta.tmp at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) at java.nio.channels.FileChannel.open(FileChannel.java:287) at java.nio.channels.FileChannel.open(FileChannel.java:335) at org.apache.cassandra.io.util.SequentialWriter.openChannel(SequentialWriter.java:113)

The tests started failing every time after CEP 14 was committed.

For reference:

• build before - https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/1484/workflows/88cac38a-ee0c-46e2-a6a7-5c364ff04f3e/jobs/9680
• build after - https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/1485/workflows/c3f56ec1-2ef0-4e61-b689-c7764934827d/jobs/9682

I think the only unrelated is 

testTokenGenerator_single_rack_or_single_rf