[CASSANDRA-10804] C* keeps compacting a keyspace when no longer exists

• Type: Bug
• Status: Open
• Priority: Low
• Resolution: Unresolved
• Fix Version/s: 2.2.x, 3.0.x, 3.11.x
• Component/s: Local/Compaction

Environment

Cassandra 2.2.3, Linux, Java 1.8.0U65, dedicated data disk

• Severity: Low
• Assignee: Unassigned
• Reporter: Francisco Madrid
• Created: 02/Dec/15 17:08
• Updated: 04/Sep/23 10:16

Description

Testing a 4 node cassandra cluster, I ran the C* stress tool but it failed with:

java.util.NoSuchElementException
   at org.apache.cassandra.stress.Operation.error(Operation.java:216)
   at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:188)
   at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:156)
   at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:309)
java.io.IOException: Operation x10 on key(s) [E>I#;+: Error executing: (NoSuchElementException)

I noticed that a compaction was running quite a time afterwards and it didn't seem to go on and stayed at 0%:

# ./bin/nodetool compactionstats
pending tasks: 1
                                     id   compaction type   keyspace   table   completed        total    unit   progress
   6bf4dbc0-990a-11e5-a703-cd0c425c8a88        Compaction       test   test1           0   1873185798   bytes      0,00%
Active compaction remaining time :   0h01m51s

I dropped the keyspace, but although the keyspace no longer existed the compaction continued:

# ./bin/cqlsh
[cqlsh 5.0.1 | Cassandra 2.2.3 | CQL spec 3.3.1 | Native protocol v4]
cqlsh> DROP KEYSPACE test;
/opt/dsc-cassandra-2.2.3/bin/../lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py:3331: DeprecationWarning: ResponseFuture.result timeout argument is deprecated. Specify the request timeout via Session.execute[_async].
cqlsh> DESC KEYSPACES;
system_traces  system_auth  system  system_distributed
cqlsh> exit

# ./bin/nodetool compactionstats
pending tasks: 1
                                     id   compaction type   keyspace   table   completed        total    unit   progress
   6bf4dbc0-990a-11e5-a703-cd0c425c8a88        Compaction       test   test1           0   1873185798   bytes      0,00%
Active compaction remaining time :   0h01m51s

I checked the data disk and there are no files in the keyspace directory.