Fix Version/s: 1.2.5
Ubuntu 10.04.1 LTS
Sun Java 6 u43
Not using VNodes
Read comment 6. This description summarizes the repair issue only, but I believe there is a bigger problem going on with networking as described on that comment.
Since I have upgraded our sandbox cluster, I am unable to run repair on any node and I am reaching our gc_grace seconds this weekend. Please help. So far, I have tried the following suggestions:
- nodetool scrub
- offline scrub
- running repair on each CF separately. Didn't matter. All got stuck the same way.
The repair command just gets stuck and the machine is idling. Only the following logs are printed for repair job:
INFO [Thread-42214] 2013-04-05 23:30:27,785 StorageService.java (line 2379) Starting repair command #4, repairing 1 ranges for keyspace cardspring_production
INFO [AntiEntropySessions:7] 2013-04-05 23:30:27,789 AntiEntropyService.java (line 652) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 new session: will sync /X.X.X.190, /X.X.X.43, /X.X.X.56 on range (1808575600,42535295865117307932921825930779602032] for keyspace_production.[comma separated list of CFs]
INFO [AntiEntropySessions:7] 2013-04-05 23:30:27,790 AntiEntropyService.java (line 858) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 requesting merkle trees for BusinessConnectionIndicesEntries (to [/X.X.X.43, /X.X.X.56, /X.X.X.190])
INFO [AntiEntropyStage:1] 2013-04-05 23:30:28,086 AntiEntropyService.java (line 214) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 Received merkle tree for ColumnFamilyName from /X.X.X.43
INFO [AntiEntropyStage:1] 2013-04-05 23:30:28,147 AntiEntropyService.java (line 214) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 Received merkle tree for ColumnFamilyName from /X.X.X.56