[CASSANDRA-13576] test failure in bootstrap_test.TestBootstrap.consistent_range_movement_false_with_rf1_should_succeed_test

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-alpha1, 4.0
• Labels: dtest test-failure
• Severity: Normal
• Assignee: Marcus Eriksson
• Reporter: Michael Hamm

Authors

Marcus Eriksson

Reviewers

Alex Petrov

• Created: 06/Jun/17 20:18
• Updated: 15/May/20 08:02
• Resolved: 17/Aug/17 14:41

Description

example failure:

http://cassci.datastax.com/job/trunk_offheap_dtest/445/testReport/bootstrap_test/TestBootstrap/consistent_range_movement_false_with_rf1_should_succeed_test

Error Message
31 May 2017 04:28:09 [node3] Missing: ['Starting listening for CQL clients']:
INFO  [main] 2017-05-31 04:18:01,615 YamlConfigura.....
See system.log for remainder

Stacktrace

  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/home/automaton/cassandra-dtest/bootstrap_test.py", line 236, in consistent_range_movement_false_with_rf1_should_succeed_test
    self._bootstrap_test_with_replica_down(False, rf=1)
  File "/home/automaton/cassandra-dtest/bootstrap_test.py", line 278, in _bootstrap_test_with_replica_down
    jvm_args=["-Dcassandra.consistent.rangemovement={}".format(consistent_range_movement)])
  File "/home/automaton/venv/local/lib/python2.7/site-packages/ccmlib/node.py", line 696, in start
    self.wait_for_binary_interface(from_mark=self.mark)
  File "/home/automaton/venv/local/lib/python2.7/site-packages/ccmlib/node.py", line 514, in wait_for_binary_interface
    self.watch_log_for("Starting listening for CQL clients", **kwargs)
  File "/home/automaton/venv/local/lib/python2.7/site-packages/ccmlib/node.py", line 471, in watch_log_for
    raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + " [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" + reads[:50] + ".....\nSee {} for remainder".format(filename))
"31 May 2017 04:28:09 [node3] Missing: ['Starting listening for CQL clients']:\nINFO  [main] 2017-05-31 04:18:01,615 YamlConfigura.....\n

-------------------- >> begin captured logging << --------------------\ndtest: DEBUG: cluster ccm directory: /tmp/dtest-PKphwD\ndtest: DEBUG: Done setting configuration options:\n{   'initial_token': None,\n    'memtable_allocation_type': 'offheap_objects',\n    'num_tokens': '32',\n    'phi_convict_threshold': 5,\n    'range_request_timeout_in_ms': 10000,\n    'read_request_timeout_in_ms': 10000,\n    'request_timeout_in_ms': 10000,\n    'truncate_request_timeout_in_ms': 10000,\n    'write_request_timeout_in_ms': 10000}\ncassandra.policies: INFO: Using datacenter 'datacenter1' for DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes\ncassandra.cluster: INFO: New Cassandra host <Host: 127.0.0.2 datacenter1> discovered\ncassandra.protocol: WARNING: Server warning: When increasing replication factor you need to run a full (-full) repair to distribute the data.\ncassandra.connection: WARNING: Heartbeat failed for connection (139927174110160) to 127.0.0.2\ncassandra.cluster: WARNING: Host 127.0.0.2 has been marked down\ncassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 2.0 seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused\ncassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 4.0 seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused\ncassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 8.0 seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused\ncassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 16.0 seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused\ncassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 32.0 seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused\ncassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 64.0 seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused\ncassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 128.0 seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused\ncassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 256.0 seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused\ncassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 512.0 seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused\n--------------------- >> end captured logging << ---------------------"

Issue Links

• duplicates: CASSANDRA-13654 bootstrap_test.TestBootstrap.consistent_range_movement_false_with_rf1_should_succeed_test consistent_range_movement_false_with_rf1_should_succeed_test
• is related to: CASSANDRA-4650 RangeStreamer should be smarter when picking endpoints for streaming in case of N >=3 in each DC.