Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14018

All dtests in bootstrap_test that call _base_bootstrap_test will cause nose to deadlock

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Normal
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Test/dtest
    • Labels:
    • Severity:
      Normal

      Description

      Any of the dtests that eventually call BaseBootstrapTest#_base_bootstrap_test will hang forever

      this includes:
      bootstrap_test: simple_bootstrap_test_with_ssl
      bootstrap_test: simple_bootstrap_test
      bootstrap_test: bootstrap_on_write_survey_test

      I've added debug logging after the tests finish their last assert and that reliably is logged.. debugging with gdb has been futile so far. The hang is on a futext inside nose during test teardown. I believe it might have something to do with the threads we spin up for the cleanup (compaction cleanup) that prevents the test cleanup from finishing because we hang waiting for the threads to exit and they never do.

      Until we get these tests fixed we should disable them as they will always fail and cause dtest runs to be very long and painful to run.

      simple_bootstrap_test_with_ssl-novnodes (bootstrap_test.TestBootstrap) ... cluster ccm directory: /tmp/dtest-ors2iY
      Done setting configuration options:

      { 'num_tokens': None, 'phi_convict_threshold': 5, 'range_request_timeout_in_ms': 10000, 'read_request_timeout_in_ms': 10000, 'request_timeout_in_ms': 10000, 'truncate_request_timeout_in_ms': 10000, 'write_request_timeout_in_ms': 10000}

      **using internode ssl**
      generating keystore.jks in [/tmp/dtest-ors2iY]
      exporting cert from keystore.jks in [/tmp/dtest-ors2iY]
      Certificate stored in file </tmp/dtest-ors2iY/ccm_node.cer>
      importing cert into truststore.jks in [/tmp/dtest-ors2iY]
      Certificate was added to keystore
      [node1, node2] tokens: [-9223372036854775808, 0]
      node1 empty size : 130.73
      node1 size before bootstrapping node2: 747.61
      node1 size after cleanup: 473.43
      node1 size after compacting: 447.91
      node2 size after compacting: 434.44

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mkjellman Michael Kjellman
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: