[CASSANDRA-15437] Decommission fails with "Unable to stream hints since no live endpoints seen" even if no hints need to be sent

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Cluster/Membership
• Labels: pull-request-available
• Bug Category: Correctness
• Severity: Low
• Complexity: Normal
• Discovered By: Adhoc Test
• Platform: All
• Assignee: Stefan Miklosovic
• Reporter: YCozy

Authors

Stefan Miklosovic

• Created: 26/Nov/19 04:04
• Updated: 16/Mar/22 12:57

Description

Dear Cassandra developers, I was applying fault-injection to test Cassandra and noticed the following behavior. I think this may be a bug. Please let me know if I'm missing something.

 

Step to reproduce:

1. Start a two node cluster (node1 & node2) using ccm.
2. Add another node to the cluster (node3).
3. Partition node3 from the other two nodes.
4. Try to decommission node3 using nodetool decommission.
5. Notice that the decommission failed with the following error log:

 

ERROR [RMI TCP Connection(4)-127.0.0.1] 2019-11-25 22:45:27,716 StorageService.java:4198 - Error while decommissioning node 
 java.lang.RuntimeException: Unable to stream hints since no live endpoints seen
  at org.apache.cassandra.service.StorageService.getPreferredHintsStreamTarget(StorageService.java:4281)
  at org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.run(HintsDispatchExecutor.java:156)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  at java.lang.Thread.run(Thread.java:748)

 

Since I didn't write any data, there is no hint to be sent. In this case, shouldn't the decommission continue?