Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-5432

Repair Freeze/Gossip Invisibility Issues 1.2.4


    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Fix Version/s: 1.2.5
    • Component/s: None
    • Labels:
    • Environment:

      Ubuntu 10.04.1 LTS
      C* 1.2.3
      Sun Java 6 u43
      JNA Enabled
      Not using VNodes


      Read comment 6. This description summarizes the repair issue only, but I believe there is a bigger problem going on with networking as described on that comment.

      Since I have upgraded our sandbox cluster, I am unable to run repair on any node and I am reaching our gc_grace seconds this weekend. Please help. So far, I have tried the following suggestions:

      • nodetool scrub
      • offline scrub
      • running repair on each CF separately. Didn't matter. All got stuck the same way.

      The repair command just gets stuck and the machine is idling. Only the following logs are printed for repair job:

      INFO [Thread-42214] 2013-04-05 23:30:27,785 StorageService.java (line 2379) Starting repair command #4, repairing 1 ranges for keyspace cardspring_production
      INFO [AntiEntropySessions:7] 2013-04-05 23:30:27,789 AntiEntropyService.java (line 652) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 new session: will sync /X.X.X.190, /X.X.X.43, /X.X.X.56 on range (1808575600,42535295865117307932921825930779602032] for keyspace_production.[comma separated list of CFs]
      INFO [AntiEntropySessions:7] 2013-04-05 23:30:27,790 AntiEntropyService.java (line 858) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 requesting merkle trees for BusinessConnectionIndicesEntries (to [/X.X.X.43, /X.X.X.56, /X.X.X.190])
      INFO [AntiEntropyStage:1] 2013-04-05 23:30:28,086 AntiEntropyService.java (line 214) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 Received merkle tree for ColumnFamilyName from /X.X.X.43
      INFO [AntiEntropyStage:1] 2013-04-05 23:30:28,147 AntiEntropyService.java (line 214) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 Received merkle tree for ColumnFamilyName from /X.X.X.56

      Please advise.


          Issue Links



              • Assignee:
                vijay2win@yahoo.com Vijay
                arya Arya Goudarzi
                Brandon Williams
              • Votes:
                1 Vote for this issue
                6 Start watching this issue


                • Created: