[CASSANDRA-6603] "hung" repair results in drain hanging

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 1.2.14

Environment

1.2.12 w/ 1.2.13 patches

• Severity: Low
• Assignee: Jason Brown
• Reporter: Chris Burroughs

Authors

Jason Brown

Reviewers

Yuki Morishita

• Created: 17/Jan/14 18:26
• Updated: 16/Apr/19 09:31
• Resolved: 17/Jan/14 23:14

Description

A "hung" repair (pile of outstanding streams with no visible progress) can result in drain never completing of run. This is a problem because restarting is a reasonable thing to do with a node that has a hung repair, and drain is a standard part of the restart procedure. I have had this happen > 20 times.

 WARN [RMI TCP Connection(7752)-10.20.6.115] 2014-01-17 12:56:51,162 StorageService.java (line 288) Stopping gossip by operator request
 INFO [RMI TCP Connection(7752)-10.20.6.115] 2014-01-17 12:56:51,162 Gossiper.java (line 1194) Announcing shutdown
 INFO [RMI TCP Connection(7754)-10.20.6.115] 2014-01-17 12:57:09,217 StorageService.java (line 942) DRAINING: starting drain process
 INFO [RMI TCP Connection(7754)-10.20.6.115] 2014-01-17 12:57:09,217 ThriftServer.java (line 116) Stop listening to thrift clients
 INFO [RMI TCP Connection(7754)-10.20.6.115] 2014-01-17 12:57:09,251 Gossiper.java (line 1194) Announcing shutdown
 INFO [RMI TCP Connection(7754)-10.20.6.115] 2014-01-17 12:57:11,252 MessagingService.java (line 694) Waiting for messaging service to quiesce
 INFO [ACCEPT-ldc1e.clearspring.local/10.20.6.115] 2014-01-17 12:57:11,253 MessagingService.java (line 904) MessagingService shutting down server thread.
 ...
wait 10 minutes with nothing happening