[CASSANDRA-14018] All dtests in bootstrap_test that call _base_bootstrap_test will cause nose to deadlock

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Auto Closed
• Component/s: Test/dtest/python
• Labels: dtest
• Severity: Normal
• Assignee: Unassigned
• Reporter: Michael Kjellman
• Created: 15/Nov/17 22:06
• Updated: 13/Sep/21 17:12
• Resolved: 13/Sep/21 17:12

Description

Any of the dtests that eventually call BaseBootstrapTest#_base_bootstrap_test will hang forever

this includes:
bootstrap_test: simple_bootstrap_test_with_ssl
bootstrap_test: simple_bootstrap_test
bootstrap_test: bootstrap_on_write_survey_test

I've added debug logging after the tests finish their last assert and that reliably is logged.. debugging with gdb has been futile so far. The hang is on a futext inside nose during test teardown. I believe it might have something to do with the threads we spin up for the cleanup (compaction cleanup) that prevents the test cleanup from finishing because we hang waiting for the threads to exit and they never do.

Until we get these tests fixed we should disable them as they will always fail and cause dtest runs to be very long and painful to run.

simple_bootstrap_test_with_ssl-novnodes (bootstrap_test.TestBootstrap) ... cluster ccm directory: /tmp/dtest-ors2iY
Done setting configuration options:

{ 'num_tokens': None, 'phi_convict_threshold': 5, 'range_request_timeout_in_ms': 10000, 'read_request_timeout_in_ms': 10000, 'request_timeout_in_ms': 10000, 'truncate_request_timeout_in_ms': 10000, 'write_request_timeout_in_ms': 10000}

**using internode ssl**
generating keystore.jks in [/tmp/dtest-ors2iY]
exporting cert from keystore.jks in [/tmp/dtest-ors2iY]
Certificate stored in file </tmp/dtest-ors2iY/ccm_node.cer>
importing cert into truststore.jks in [/tmp/dtest-ors2iY]
Certificate was added to keystore
[node1, node2] tokens: [-9223372036854775808, 0]
node1 empty size : 130.73
node1 size before bootstrapping node2: 747.61
node1 size after cleanup: 473.43
node1 size after compacting: 447.91
node2 size after compacting: 434.44