Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-6603

"hung" repair results in drain hanging

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Low
    • Resolution: Fixed
    • 1.2.14
    • None
    • None
    • 1.2.12 w/ 1.2.13 patches

    • Low

    Description

      A "hung" repair (pile of outstanding streams with no visible progress) can result in drain never completing of run. This is a problem because restarting is a reasonable thing to do with a node that has a hung repair, and drain is a standard part of the restart procedure. I have had this happen > 20 times.

       WARN [RMI TCP Connection(7752)-10.20.6.115] 2014-01-17 12:56:51,162 StorageService.java (line 288) Stopping gossip by operator request
       INFO [RMI TCP Connection(7752)-10.20.6.115] 2014-01-17 12:56:51,162 Gossiper.java (line 1194) Announcing shutdown
       INFO [RMI TCP Connection(7754)-10.20.6.115] 2014-01-17 12:57:09,217 StorageService.java (line 942) DRAINING: starting drain process
       INFO [RMI TCP Connection(7754)-10.20.6.115] 2014-01-17 12:57:09,217 ThriftServer.java (line 116) Stop listening to thrift clients
       INFO [RMI TCP Connection(7754)-10.20.6.115] 2014-01-17 12:57:09,251 Gossiper.java (line 1194) Announcing shutdown
       INFO [RMI TCP Connection(7754)-10.20.6.115] 2014-01-17 12:57:11,252 MessagingService.java (line 694) Waiting for messaging service to quiesce
       INFO [ACCEPT-ldc1e.clearspring.local/10.20.6.115] 2014-01-17 12:57:11,253 MessagingService.java (line 904) MessagingService shutting down server thread.
       ...
      wait 10 minutes with nothing happening
      

      Attachments

        1. 6603_v1.diff
          2 kB
          Jason Brown
        2. drain.stack
          6 kB
          Chris Burroughs
        3. CassandraDaemon.stack2
          307 kB
          Chris Burroughs
        4. CassandraDaemon.stack
          307 kB
          Chris Burroughs

        Activity

          People

            jasobrown Jason Brown
            cburroughs Chris Burroughs
            Jason Brown
            Yuki Morishita
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: