[CASSANDRA-18361] Test Failure: secondary_indexes_test.py::TestSecondaryIndexes::test_failing_manual_rebuild_index

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0.12, 4.1.4
• Component/s: Test/dtest/python
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: DTest
• Platform: All
• Since Version: 4.0.x

Source Control Link

https://github.com/apache/cassandra/commit/016d91a7d7d6c7998d1f3cb35726baf8dd9bac03, https://github.com/apache/cassandra-dtest/commit/3af85373d70fb5d549447f6520da1d11a228d71a

Test and Documentation Plan

run repeated tests

• Assignee: Jacek Lewandowski
• Reporter: Andres de la Peña

Authors

Jacek Lewandowski

Reviewers

Andres de la Peña, Piotr Kolaczkowski

• Created: 24/Mar/23 12:49
• Updated: 05/Oct/23 13:46
• Resolved: 05/Oct/23 13:39

Description

The Python dtest secondary_indexes_test.py::TestSecondaryIndexes::test_failing_manual_rebuild_index is flaky, at least for trunk:

• https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/secondary_indexes_test/TestSecondaryIndexes/test_failing_manual_rebuild_index
• https://ci-cassandra.apache.org/job/Cassandra-trunk/1501/testReport/dtest.secondary_indexes_test/TestSecondaryIndexes/test_failing_manual_rebuild_index/

Error Message
failed on teardown with "Unexpected error found in node logs (see stdout for full details). Errors: [[node1] 'ERROR [Reference-Reaper] 2023-03-23 00:23:43,597 Ref.java:237 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@967019010:/home/cassandra/cassandra/cassandra-dtest/tmp/dtest-hgjoy8rq/test/node1/data0/k/t-b7dae870c91011eda58f05bc40bfcaa1/nc-1-big-Index.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@967019010:/home/cassandra/cassandra/cassandra-dtest/tmp/dtest-hgjoy8rq/test/node1/data0/k/t-b7dae870c91011eda58f05bc40bfcaa1/nc-1-big-Index.db was not released before the reference was garbage collected']"
Stacktrace
Unexpected error found in node logs (see stdout for full details). Errors: [[node1] 'ERROR [Reference-Reaper] 2023-03-23 00:23:43,597 Ref.java:237 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@967019010:/home/cassandra/cassandra/cassandra-dtest/tmp/dtest-hgjoy8rq/test/node1/data0/k/t-b7dae870c91011eda58f05bc40bfcaa1/nc-1-big-Index.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@967019010:/home/cassandra/cassandra/cassandra-dtest/tmp/dtest-hgjoy8rq/test/node1/data0/k/t-b7dae870c91011eda58f05bc40bfcaa1/nc-1-big-Index.db was not released before the reference was garbage collected']

The failure can be reproduced in CircleCI:

• https://app.circleci.com/pipelines/github/adelapena/cassandra/2732/workflows/829434ab-2d1a-4e1c-8c7f-42449fcfda22

The CircleCI config I used to reproduce the test failure can be generated with:

.circleci/generate.sh -p \
  -e REPEATED_DTESTS_COUNT=200 \
  -e REPEATED_DTESTS=secondary_indexes_test.py::TestSecondaryIndexes::test_failing_manual_rebuild_index

Issue Links

• is duplicated by: CASSANDRA-18522 LEAK DETECTED: org.apache.cassandra.io.util.FileHandle$Cleanup in TestSecondaryIndexes.test_failing_manual_rebuild_index
• links to: GitHub Pull Request #2749
• links to: GitHub Pull Request #2751