[CASSANDRA-11539] dtest failure in topology_test.TestTopology.movement_test

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.11.5
• Component/s: Test/dtest/python
• Labels: dtest
• Assignee: Russ Hatch
• Reporter: Michael Shuler

Authors

Russ Hatch

• Created: 08/Apr/16 20:15
• Updated: 01/Aug/21 11:28
• Resolved: 27/Apr/16 18:51

Description

example failure:

Error Message

values not within 16.00% of the max: (335.88, 404.31) ()
-------------------- >> begin captured logging << --------------------
dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-XGOyDd
dtest: DEBUG: Custom init_config not found. Setting defaults.
dtest: DEBUG: Done setting configuration options:
{   'num_tokens': None,
    'phi_convict_threshold': 5,
    'range_request_timeout_in_ms': 10000,
    'read_request_timeout_in_ms': 10000,
    'request_timeout_in_ms': 10000,
    'truncate_request_timeout_in_ms': 10000,
    'write_request_timeout_in_ms': 10000}
--------------------- >> end captured logging << ---------------------
Stacktrace

  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/home/automaton/cassandra-dtest/topology_test.py", line 93, in movement_test
    assert_almost_equal(sizes[1], sizes[2])
  File "/home/automaton/cassandra-dtest/assertions.py", line 75, in assert_almost_equal
    assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within %.2f%% of the max: %s (%s)" % (error * 100, args, error_message)
"values not within 16.00% of the max: (335.88, 404.31) ()\n-------------------- >> begin captured logging << --------------------\ndtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-XGOyDd\ndtest: DEBUG: Custom init_config not found. Setting defaults.\ndtest: DEBUG: Done setting configuration options:\n{   'num_tokens': None,\n    'phi_convict_threshold': 5,\n    'range_request_timeout_in_ms': 10000,\n    'read_request_timeout_in_ms': 10000,\n    'request_timeout_in_ms': 10000,\n    'truncate_request_timeout_in_ms': 10000,\n    'write_request_timeout_in_ms': 10000}\n--------------------- >> end captured logging << ---------------------"

http://cassci.datastax.com/job/cassandra-3.5_novnode_dtest/22/testReport/topology_test/TestTopology/movement_test

---

I dug through this test's history on the trunk, 3.5, 3.0, and 2.2 branches. It appears this test is stable and passing on 3.0 & 2.2 (which could be just luck). On trunk & 3.5, however, this test has flapped a small number of times.

The test's threshold is 16% and I found test failures in the 3.5 branch of 16.2%, 16.9%, and 18.3%. In trunk I found 17.4% and 23.5% diff failures.

Issue Links

• is duplicated by: CASSANDRA-11666 dtest failure in topology_test.TestTopology.movement_test