[CASSANDRA-7005] repair_test dtest fails on 2.1

• Type: Improvement
• Status: Resolved
• Priority: Urgent
• Resolution: Duplicate
• Fix Version/s: 2.1 beta2
• Component/s: Test/dtest/python
• Labels: qa-resolved
• Assignee: Yuki Morishita
• Reporter: Michael Shuler

Authors

Yuki Morishita

• Tester: Michael Shuler
• Created: 08/Apr/14 18:50
• Updated: 16/Apr/19 09:31
• Resolved: 11/Apr/14 21:44

Description

$ PRINT_DEBUG=true nosetests --nocapture --nologcapture --verbosity=3 repair_test.py
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
simple_repair_order_preserving_test (repair_test.TestRepair) ... cluster ccm directory: /tmp/dtest-BVfye7
Starting cluster..
Inserting data...
Checking data on node3...
Checking data on node1...
Checking data on node2...
starting repair...
[2014-04-08 13:44:31,424] Starting repair command #1, repairing 3 ranges for keyspace ks (seq=true, full=true)
[2014-04-08 13:44:32,748] Repair session d262e390-bf4d-11e3-a482-75998baadb41 for range (00,0113427455640312821154458202477256070484] failed with error org.apache.cassandra.exceptions.RepairException: [repair #d262e390-bf4d-11e3-a482-75998baadb41 on ks/cf, (00,0113427455640312821154458202477256070484]] Validation failed in /127.0.0.2
[2014-04-08 13:44:32,751] Repair session d2b98f10-bf4d-11e3-a482-75998baadb41 for range (0113427455640312821154458202477256070484,56713727820156410577229101238628035242] failed with error org.apache.cassandra.exceptions.RepairException: [repair #d2b98f10-bf4d-11e3-a482-75998baadb41 on ks/cf, (0113427455640312821154458202477256070484,56713727820156410577229101238628035242]] Validation failed in /127.0.0.2
[2014-04-08 13:44:32,753] Repair session d2dca770-bf4d-11e3-a482-75998baadb41 for range (56713727820156410577229101238628035242,00] failed with error org.apache.cassandra.exceptions.RepairException: [repair #d2dca770-bf4d-11e3-a482-75998baadb41 on ks/cf, (56713727820156410577229101238628035242,00]] Validation failed in /127.0.0.2
[2014-04-08 13:44:32,753] Repair command #1 finished
[2014-04-08 13:44:32,770] Nothing to repair for keyspace 'system'
[2014-04-08 13:44:32,783] Starting repair command #2, repairing 2 ranges for keyspace system_traces (seq=true, full=true)
[2014-04-08 13:44:34,635] Repair session d3310900-bf4d-11e3-a482-75998baadb41 for range (0113427455640312821154458202477256070484,56713727820156410577229101238628035242] finished
[2014-04-08 13:44:34,640] Repair session d3f80280-bf4d-11e3-a482-75998baadb41 for range (56713727820156410577229101238628035242,00] finished
[2014-04-08 13:44:34,640] Repair command #2 finished
Repair time: 4.63053512573
FAIL
ERROR
simple_repair_test (repair_test.TestRepair) ... cluster ccm directory: /tmp/dtest-_L5lTP
Starting cluster..
Inserting data...
Checking data on node3...
Checking data on node1...
Checking data on node2...
starting repair...
[2014-04-08 13:47:14,109] Starting repair command #1, repairing 3 ranges for keyspace ks (seq=true, full=true)
[2014-04-08 13:47:15,291] Repair session 335a5840-bf4e-11e3-b691-75998baadb41 for range (-3074457345618258603,3074457345618258602] failed with error org.apache.cassandra.exceptions.RepairException: [repair #335a5840-bf4e-11e3-b691-75998baadb41 on ks/cf, (-3074457345618258603,3074457345618258602]] Validation failed in /127.0.0.2
[2014-04-08 13:47:15,292] Repair session 33ad0c20-bf4e-11e3-b691-75998baadb41 for range (-9223372036854775808,-3074457345618258603] failed with error org.apache.cassandra.exceptions.RepairException: [repair #33ad0c20-bf4e-11e3-b691-75998baadb41 on ks/cf, (-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
[2014-04-08 13:47:15,295] Repair session 33e978e0-bf4e-11e3-b691-75998baadb41 for range (3074457345618258602,-9223372036854775808] failed with error org.apache.cassandra.exceptions.RepairException: [repair #33e978e0-bf4e-11e3-b691-75998baadb41 on ks/cf, (3074457345618258602,-9223372036854775808]] Validation failed in /127.0.0.2
[2014-04-08 13:47:15,295] Repair command #1 finished
[2014-04-08 13:47:15,307] Nothing to repair for keyspace 'system'
[2014-04-08 13:47:15,322] Starting repair command #2, repairing 2 ranges for keyspace system_traces (seq=true, full=true)
[2014-04-08 13:47:15,983] Repair session 3412f9e0-bf4e-11e3-b691-75998baadb41 for range (-3074457345618258603,3074457345618258602] finished
[2014-04-08 13:47:15,988] Repair session 345d9770-bf4e-11e3-b691-75998baadb41 for range (3074457345618258602,-9223372036854775808] finished
[2014-04-08 13:47:15,988] Repair command #2 finished
Repair time: 3.25424218178
FAIL
ERROR

======================================================================
ERROR: simple_repair_order_preserving_test (repair_test.TestRepair)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mshuler/git/cassandra-dtest/dtest.py", line 202, in tearDown
    raise AssertionError('Unexpected error in %s node log: %s' % (node.name, errors))
AssertionError: Unexpected error in node1 node log: ['ERROR [AntiEntropySessions:1] 2014-04-08 13:44:32,011 RepairSession.java:289 - [repair #d262e390-bf4d-11e3-a482-75998baadb41] session completed with the following error\n', 'ERROR [AntiEntropySessions:1] 2014-04-08 13:44:32,013 CassandraDaemon.java:166 - Exception in thread Thread[AntiEntropySessions:1,5,RMI Runtime]\n', 'ERROR [AntiEntropySessions:2] 2014-04-08 13:44:32,247 RepairSession.java:289 - [repair #d2b98f10-bf4d-11e3-a482-75998baadb41] session completed with the following error\n', 'ERROR [AntiEntropySessions:2] 2014-04-08 13:44:32,249 CassandraDaemon.java:166 - Exception in thread Thread[AntiEntropySessions:2,5,RMI Runtime]\n', 'ERROR [Thread-8] 2014-04-08 13:44:32,747 StorageService.java:2659 - Repair session d262e390-bf4d-11e3-a482-75998baadb41 for range (00,0113427455640312821154458202477256070484] failed with error org.apache.cassandra.exceptions.RepairException: [repair #d262e390-bf4d-11e3-a482-75998baadb41 on ks/cf, (00,0113427455640312821154458202477256070484]] Validation failed in /127.0.0.2\n', 'ERROR [AntiEntropySessions:3] 2014-04-08 13:44:32,749 RepairSession.java:289 - [repair #d2dca770-bf4d-11e3-a482-75998baadb41] session completed with the following error\n', 'ERROR [AntiEntropySessions:3] 2014-04-08 13:44:32,750 CassandraDaemon.java:166 - Exception in thread Thread[AntiEntropySessions:3,5,RMI Runtime]\n', 'ERROR [Thread-8] 2014-04-08 13:44:32,751 StorageService.java:2659 - Repair session d2b98f10-bf4d-11e3-a482-75998baadb41 for range (0113427455640312821154458202477256070484,56713727820156410577229101238628035242] failed with error org.apache.cassandra.exceptions.RepairException: [repair #d2b98f10-bf4d-11e3-a482-75998baadb41 on ks/cf, (0113427455640312821154458202477256070484,56713727820156410577229101238628035242]] Validation failed in /127.0.0.2\n', 'ERROR [Thread-8] 2014-04-08 13:44:32,752 StorageService.java:2659 - Repair session d2dca770-bf4d-11e3-a482-75998baadb41 for range (56713727820156410577229101238628035242,00] failed with error org.apache.cassandra.exceptions.RepairException: [repair #d2dca770-bf4d-11e3-a482-75998baadb41 on ks/cf, (56713727820156410577229101238628035242,00]] Validation failed in /127.0.0.2\n']

======================================================================
ERROR: simple_repair_test (repair_test.TestRepair)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mshuler/git/cassandra-dtest/dtest.py", line 202, in tearDown
    raise AssertionError('Unexpected error in %s node log: %s' % (node.name, errors))
AssertionError: Unexpected error in node1 node log: ['ERROR [AntiEntropySessions:1] 2014-04-08 13:47:14,712 RepairSession.java:289 - [repair #335a5840-bf4e-11e3-b691-75998baadb41] session completed with the following error\n', 'ERROR [AntiEntropySessions:1] 2014-04-08 13:47:14,714 CassandraDaemon.java:166 - Exception in thread Thread[AntiEntropySessions:1,5,RMI Runtime]\n', 'ERROR [AntiEntropySessions:2] 2014-04-08 13:47:15,058 RepairSession.java:289 - [repair #33ad0c20-bf4e-11e3-b691-75998baadb41] session completed with the following error\n', 'ERROR [AntiEntropySessions:2] 2014-04-08 13:47:15,058 CassandraDaemon.java:166 - Exception in thread Thread[AntiEntropySessions:2,5,RMI Runtime]\n', 'ERROR [AntiEntropySessions:3] 2014-04-08 13:47:15,288 RepairSession.java:289 - [repair #33e978e0-bf4e-11e3-b691-75998baadb41] session completed with the following error\n', 'ERROR [AntiEntropySessions:3] 2014-04-08 13:47:15,289 CassandraDaemon.java:166 - Exception in thread Thread[AntiEntropySessions:3,5,RMI Runtime]\n', 'ERROR [Thread-8] 2014-04-08 13:47:15,290 StorageService.java:2659 - Repair session 335a5840-bf4e-11e3-b691-75998baadb41 for range (-3074457345618258603,3074457345618258602] failed with error org.apache.cassandra.exceptions.RepairException: [repair #335a5840-bf4e-11e3-b691-75998baadb41 on ks/cf, (-3074457345618258603,3074457345618258602]] Validation failed in /127.0.0.2\n', 'ERROR [Thread-8] 2014-04-08 13:47:15,291 StorageService.java:2659 - Repair session 33ad0c20-bf4e-11e3-b691-75998baadb41 for range (-9223372036854775808,-3074457345618258603] failed with error org.apache.cassandra.exceptions.RepairException: [repair #33ad0c20-bf4e-11e3-b691-75998baadb41 on ks/cf, (-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2\n', 'ERROR [Thread-8] 2014-04-08 13:47:15,294 StorageService.java:2659 - Repair session 33e978e0-bf4e-11e3-b691-75998baadb41 for range (3074457345618258602,-9223372036854775808] failed with error org.apache.cassandra.exceptions.RepairException: [repair #33e978e0-bf4e-11e3-b691-75998baadb41 on ks/cf, (3074457345618258602,-9223372036854775808]] Validation failed in /127.0.0.2\n']

======================================================================
FAIL: simple_repair_order_preserving_test (repair_test.TestRepair)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mshuler/git/cassandra-dtest/repair_test.py", line 39, in simple_repair_order_preserving_test
    self._simple_repair(order_preserving_partitioner=True)
  File "/home/mshuler/git/cassandra-dtest/repair_test.py", line 95, in _simple_repair
    assert len(l) == 2, "Lines matching: " + str([elt[0] for elt in l])
AssertionError: Lines matching: []

======================================================================
FAIL: simple_repair_test (repair_test.TestRepair)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mshuler/git/cassandra-dtest/repair_test.py", line 36, in simple_repair_test
    self._simple_repair()
  File "/home/mshuler/git/cassandra-dtest/repair_test.py", line 95, in _simple_repair
    assert len(l) == 2, "Lines matching: " + str([elt[0] for elt in l])
AssertionError: Lines matching: []

----------------------------------------------------------------------
Ran 2 tests in 324.740s

FAILED (errors=2, failures=2)

Issue Links

• duplicates: CASSANDRA-7000 Assertion in SSTableReader during repair.