[CASSANDRA-11824] If repair fails no way to run repair again

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1.15, 2.2.7, 3.0.7, 3.7
• Labels: fallout
• Severity: Normal
• Assignee: Marcus Eriksson Assign to me
• Reporter: T Jake Luciani

Authors

Marcus Eriksson

Reviewers

Paulo Motta

• Created: 18/May/16 01:13
• Updated: 16/Apr/19 09:30
• Resolved: 26/May/16 06:38

Description

I have a test that disables gossip and runs repair at the same time.

WARN [RMI TCP Connection(15)-54.67.121.105] 2016-05-17 16:57:21,775 StorageService.java:384 - Stopping gossip by operator request
INFO [RMI TCP Connection(15)-54.67.121.105] 2016-05-17 16:57:21,775 Gossiper.java:1463 - Announcing shutdown
INFO [RMI TCP Connection(15)-54.67.121.105] 2016-05-17 16:57:21,776 StorageService.java:1999 - Node /172.31.31.1 state jump to shutdown
INFO [HANDSHAKE-/172.31.17.32] 2016-05-17 16:57:21,895 OutboundTcpConnection.java:514 - Handshaking version with /172.31.17.32
INFO [HANDSHAKE-/172.31.24.76] 2016-05-17 16:57:21,895 OutboundTcpConnection.java:514 - Handshaking version with /172.31.24.76
INFO [Thread-25] 2016-05-17 16:57:21,925 RepairRunnable.java:125 - Starting repair command #1, repairing keyspace keyspace1 with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 3)
INFO [Thread-26] 2016-05-17 16:57:21,953 RepairRunnable.java:125 - Starting repair command #2, repairing keyspace stresscql with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 3)
INFO [Thread-27] 2016-05-17 16:57:21,967 RepairRunnable.java:125 - Starting repair command #3, repairing keyspace system_traces with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 2)

This ends up failing:

16:54:44.844 INFO serverGroup-node-1-574 - STDOUT: [2016-05-17 16:57:21,933] Starting repair command #1, repairing keyspace keyspace1 with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 3)
[2016-05-17 16:57:21,943] Did not get positive replies from all endpoints. List of failed endpoint(s): [172.31.24.76, 172.31.17.32]
[2016-05-17 16:57:21,945] null

Subsequent calls to repair with all nodes up still fails:

ERROR [ValidationExecutor:3] 2016-05-17 18:58:53,460 CompactionManager.java:1193 - Cannot start multiple repair sessions over the same sstables
ERROR [ValidationExecutor:3] 2016-05-17 18:58:53,460 Validator.java:261 - Failed creating a merkle tree for [repair #66425f10-1c61-11e6-83b2-0b1fff7a067d on keyspace1/standard1,

Issue Links

• is duplicated by: CASSANDRA-12860 Nodetool repair fragile: cannot properly recover from single node failure. Has to restart all nodes in order to repair again