[CASSANDRA-5432] Repair Freeze/Gossip Invisibility Issues 1.2.4

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 1.2.5

Environment

Ubuntu 10.04.1 LTS
C* 1.2.3
Sun Java 6 u43
JNA Enabled
Not using VNodes

• Severity: Critical
• Assignee: Vijay
• Reporter: Arya Goudarzi

Authors

Vijay

Reviewers

Brandon Williams

• Created: 06/Apr/13 01:02
• Updated: 16/Apr/19 09:32
• Resolved: 06/May/13 07:16

Description

Read comment 6. This description summarizes the repair issue only, but I believe there is a bigger problem going on with networking as described on that comment.

Since I have upgraded our sandbox cluster, I am unable to run repair on any node and I am reaching our gc_grace seconds this weekend. Please help. So far, I have tried the following suggestions:

• nodetool scrub
• offline scrub
• running repair on each CF separately. Didn't matter. All got stuck the same way.

The repair command just gets stuck and the machine is idling. Only the following logs are printed for repair job:

INFO [Thread-42214] 2013-04-05 23:30:27,785 StorageService.java (line 2379) Starting repair command #4, repairing 1 ranges for keyspace cardspring_production
INFO [AntiEntropySessions:7] 2013-04-05 23:30:27,789 AntiEntropyService.java (line 652) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 new session: will sync /X.X.X.190, /X.X.X.43, /X.X.X.56 on range (1808575600,42535295865117307932921825930779602032] for keyspace_production.[comma separated list of CFs]
INFO [AntiEntropySessions:7] 2013-04-05 23:30:27,790 AntiEntropyService.java (line 858) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 requesting merkle trees for BusinessConnectionIndicesEntries (to [/X.X.X.43, /X.X.X.56, /X.X.X.190])
INFO [AntiEntropyStage:1] 2013-04-05 23:30:28,086 AntiEntropyService.java (line 214) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 Received merkle tree for ColumnFamilyName from /X.X.X.43
INFO [AntiEntropyStage:1] 2013-04-05 23:30:28,147 AntiEntropyService.java (line 214) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 Received merkle tree for ColumnFamilyName from /X.X.X.56

Please advise.

Issue Links

• is duplicated by: CASSANDRA-5332 1.2.3 Nodetool reports 1.1.10 nodes as down
• is related to: CASSANDRA-5493 Confusing output of CommandDroppedTasks