[CASSANDRA-18747] Test failure: Fix assertion error AssertionError: Unknown keyspace system_auth\n\tat org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324)\n\tat org.apache.cassandra.db.Keyspace.lambda$open$0(Keyspace.java:162)

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.1.4, 5.0-alpha2, 5.0, 5.1
• Component/s: Cluster/Schema, Test/dtest/python
• Complexity: Normal
• Platform: All
• Since Version: 4.1.0

Source Control Link

https://github.com/apache/cassandra/commit/ac71d0f56efda081cf3c602eae8897b64cf84dac

Test and Documentation Plan

run regression tests

• Assignee: Jacek Lewandowski
• Reporter: Ekaterina Dimitrova

Authors

Jacek Lewandowski

Reviewers

Benjamin Lerer, Berenguer Blasi, Ekaterina Dimitrova, Jeremiah Jordan

• Created: 11/Aug/23 19:17
• Updated: 08/Nov/23 16:35
• Resolved: 19/Oct/23 11:16

Description

I've been seeing this assertion error in different tests lately.

Full error message:

failed on teardown with "Unexpected error found in node logs (see stdout for full details). Errors: [[node2] 'ERROR [PendingRangeCalculator:1] 2023-08-11 16:35:14,445 JVMStabilityInspector.java:70 - Exception in thread Thread[PendingRangeCalculator:1,5,PendingRangeCalculator]\njava.lang.AssertionError: Unknown keyspace system_auth\n\tat org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324)\n\tat org.apache.cassandra.db.Keyspace.lambda$open$0(Keyspace.java:162)\n\tat org.apache.cassandra.utils.concurrent.LoadingMap.blockingLoadIfAbsent(LoadingMap.java:105)\n\tat org.apache.cassandra.schema.Schema.maybeAddKeyspaceInstance(Schema.java:251)\n\tat org.apache.cassandra.db.Keyspace.open(Keyspace.java:162)\n\tat org.apache.cassandra.db.Keyspace.open(Keyspace.java:151)\n\tat org.apache.cassandra.service.PendingRangeCalculatorService.lambda$new$1(PendingRangeCalculatorService.java:58)\n\tat org.apache.cassandra.concurrent.SingleThreadExecutorPlus$AtLeastOnce.run(SingleThreadExecutorPlus.java:60)\n\tat org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:829)']" Unexpected error found in node logs (see stdout for full details). Errors: [[node2] 'ERROR [PendingRangeCalculator:1] 2023-08-11 16:35:14,445 JVMStabilityInspector.java:70 - Exception in thread Thread[PendingRangeCalculator:1,5,PendingRangeCalculator]\njava.lang.AssertionError: Unknown keyspace system_auth\n\tat org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324)\n\tat org.apache.cassandra.db.Keyspace.lambda$open$0(Keyspace.java:162)\n\tat org.apache.cassandra.utils.concurrent.LoadingMap.blockingLoadIfAbsent(LoadingMap.java:105)\n\tat org.apache.cassandra.schema.Schema.maybeAddKeyspaceInstance(Schema.java:251)\n\tat org.apache.cassandra.db.Keyspace.open(Keyspace.java:162)\n\tat org.apache.cassandra.db.Keyspace.open(Keyspace.java:151)\n\tat org.apache.cassandra.service.PendingRangeCalculatorService.lambda$new$1(PendingRangeCalculatorService.java:58)\n\tat org.apache.cassandra.concurrent.SingleThreadExecutorPlus$AtLeastOnce.run(SingleThreadExecutorPlus.java:60)\n\tat org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:829)']

Example failures:
test_failed_snitch_update_property_file_snitch - https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2475/workflows/2086619e-0f21-464b-a866-84aca516b5e5/jobs/36716/tests
test_gcgs_validation - https://ci-cassandra.apache.org/job/Cassandra-trunk/1666/testReport/junit/dtest.materialized_views_test/TestMaterializedViews/test_gcgs_validation/

Issue Links

• causes: CASSANDRA-18963 Test Failure: sstableutil_test.TestSSTableUtil
• Discovered while testing: CASSANDRA-18708 Test failure: org.apache.cassandra.tools.StandaloneSplitterWithCQLTesterTest.testNoSnapshotOption-.jdk17
• duplicates: CASSANDRA-18949 Test failure: org.apache.cassandra.tools.nodetool.ClearSnapshotTest.testClearSnapshot_RemoveByName-.jdk11.arch=x86_64.python2.7
• Testing discovered: CASSANDRA-18921 Describe statement may include inconsistent schema and schema version for paging
• links to: GitHub Pull Request #2713
• links to: GitHub Pull Request #2761
• links to: GitHub Pull Request #2763