Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-5432

Repair Freeze/Gossip Invisibility Issues 1.2.4

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Fix Version/s: 1.2.5
    • Component/s: None
    • Labels:
      None
    • Environment:

      Ubuntu 10.04.1 LTS
      C* 1.2.3
      Sun Java 6 u43
      JNA Enabled
      Not using VNodes

      Description

      Read comment 6. This description summarizes the repair issue only, but I believe there is a bigger problem going on with networking as described on that comment.

      Since I have upgraded our sandbox cluster, I am unable to run repair on any node and I am reaching our gc_grace seconds this weekend. Please help. So far, I have tried the following suggestions:

      • nodetool scrub
      • offline scrub
      • running repair on each CF separately. Didn't matter. All got stuck the same way.

      The repair command just gets stuck and the machine is idling. Only the following logs are printed for repair job:

      INFO [Thread-42214] 2013-04-05 23:30:27,785 StorageService.java (line 2379) Starting repair command #4, repairing 1 ranges for keyspace cardspring_production
      INFO [AntiEntropySessions:7] 2013-04-05 23:30:27,789 AntiEntropyService.java (line 652) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 new session: will sync /X.X.X.190, /X.X.X.43, /X.X.X.56 on range (1808575600,42535295865117307932921825930779602032] for keyspace_production.[comma separated list of CFs]
      INFO [AntiEntropySessions:7] 2013-04-05 23:30:27,790 AntiEntropyService.java (line 858) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 requesting merkle trees for BusinessConnectionIndicesEntries (to [/X.X.X.43, /X.X.X.56, /X.X.X.190])
      INFO [AntiEntropyStage:1] 2013-04-05 23:30:28,086 AntiEntropyService.java (line 214) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 Received merkle tree for ColumnFamilyName from /X.X.X.43
      INFO [AntiEntropyStage:1] 2013-04-05 23:30:28,147 AntiEntropyService.java (line 214) repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242 Received merkle tree for ColumnFamilyName from /X.X.X.56

      Please advise.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vijay2win@yahoo.com Vijay
                Reporter:
                arya Arya Goudarzi
                Reviewer:
                Brandon Williams
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: