[CASSANDRA-11499] dtest failure in commitlog_test.TestCommitLog.test_commitlog_replay_on_startup

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Cannot Reproduce
• Labels: dtest
• Assignee: Russ Hatch
• Reporter: Michael Shuler

Authors

Russ Hatch

• Created: 05/Apr/16 01:18
• Updated: 16/Apr/19 09:30
• Resolved: 22/Apr/16 18:42

Description

example failure:

http://cassci.datastax.com/job/trunk_novnode_dtest/341/testReport/commitlog_test/TestCommitLog/test_commitlog_replay_on_startup

Failed on CassCI build trunk_novnode_dtest #341

Error Message

03 Apr 2016 16:32:49 [node1] Missing: ['Log replay complete']:
INFO  [main] 2016-04-03 16:22:39,826 YamlConfigura.....
See system.log for remainder
-------------------- >> begin captured logging << --------------------
dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-1UTelU
dtest: DEBUG: Custom init_config not found. Setting defaults.
dtest: DEBUG: Done setting configuration options:
{   'num_tokens': None, 'phi_convict_threshold': 5, 'start_rpc': 'true'}
dtest: DEBUG: Insert data
dtest: DEBUG: Verify data is present
dtest: DEBUG: Stop node abruptly
dtest: DEBUG: Verify commitlog was written before abrupt stop
dtest: DEBUG: Verify no SSTables were flushed before abrupt stop
dtest: DEBUG: Verify commit log was replayed on startup
--------------------- >> end captured logging << ---------------------
Stacktrace

  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/home/automaton/cassandra-dtest/commitlog_test.py", line 193, in test_commitlog_replay_on_startup
    node1.watch_log_for("Log replay complete")
  File "/home/automaton/ccm/ccmlib/node.py", line 425, in watch_log_for
    raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + " [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" + reads[:50] + ".....\nSee {} for remainder".format(filename))
"03 Apr 2016 16:32:49 [node1] Missing: ['Log replay complete']:\nINFO  [main] 2016-04-03 16:22:39,826 YamlConfigura.....\nSee system.log for remainder\n-------------------- >> begin captured logging << --------------------\ndtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-1UTelU\ndtest: DEBUG: Custom init_config not found. Setting defaults.\ndtest: DEBUG: Done setting configuration options:\n{   'num_tokens': None, 'phi_convict_threshold': 5, 'start_rpc': 'true'}\ndtest: DEBUG: Insert data\ndtest: DEBUG: Verify data is present\ndtest: DEBUG: Stop node abruptly\ndtest: DEBUG: Verify commitlog was written before abrupt stop\ndtest: DEBUG: Verify no SSTables were flushed before abrupt stop\ndtest: DEBUG: Verify commit log was replayed on startup\n--------------------- >> end captured logging << ---------------------"