[CASSANDRA-10992] Hanging streaming sessions

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.2.8, 3.0.9, 3.8
• Component/s: Legacy/Streaming and Messaging

Environment

C* 2.1.12, Debian Wheezy

• Severity: Normal
• Assignee: Paulo Motta
• Reporter: mlowicki

Authors

Paulo Motta

Reviewers

Yuki Morishita

• Created: 09/Jan/16 17:53
• Updated: 16/Apr/19 09:30
• Resolved: 09/Aug/16 22:05

Description

I've started recently running repair using Cassandra Reaper (built-in nodetool repair doesn't work for me - CASSANDRA-9935). It behaves fine but I've noticed hanging streaming sessions:

root@db1:~# date
Sat Jan  9 16:43:00 UTC 2016
root@db1:~# nt netstats -H | grep total
        Receiving 5 files, 46.59 MB total. Already received 1 files, 11.32 MB total
        Sending 7 files, 46.28 MB total. Already sent 7 files, 46.28 MB total
        Receiving 6 files, 64.15 MB total. Already received 1 files, 12.14 MB total
        Sending 5 files, 61.15 MB total. Already sent 5 files, 61.15 MB total
        Receiving 4 files, 7.75 MB total. Already received 3 files, 7.58 MB total
        Sending 4 files, 4.29 MB total. Already sent 4 files, 4.29 MB total
        Receiving 12 files, 13.79 MB total. Already received 11 files, 7.66 MB total
        Sending 5 files, 15.32 MB total. Already sent 5 files, 15.32 MB total
        Receiving 8 files, 20.35 MB total. Already received 1 files, 13.63 MB total
        Sending 38 files, 125.34 MB total. Already sent 38 files, 125.34 MB total

root@db1:~# date
Sat Jan  9 17:45:42 UTC 2016
root@db1:~# nt netstats -H | grep total
        Receiving 5 files, 46.59 MB total. Already received 1 files, 11.32 MB total
        Sending 7 files, 46.28 MB total. Already sent 7 files, 46.28 MB total
        Receiving 6 files, 64.15 MB total. Already received 1 files, 12.14 MB total
        Sending 5 files, 61.15 MB total. Already sent 5 files, 61.15 MB total
        Receiving 4 files, 7.75 MB total. Already received 3 files, 7.58 MB total
        Sending 4 files, 4.29 MB total. Already sent 4 files, 4.29 MB total
        Receiving 12 files, 13.79 MB total. Already received 11 files, 7.66 MB total
        Sending 5 files, 15.32 MB total. Already sent 5 files, 15.32 MB total
        Receiving 8 files, 20.35 MB total. Already received 1 files, 13.63 MB total
        Sending 38 files, 125.34 MB total. Already sent 38 files, 125.34 MB total

Such sessions are left even when repair job is long time done (confirmed by checking Reaper's and Cassandra's logs). streaming_socket_timeout_in_ms in cassandra.yaml is set to default value (3600000).

Issue Links

• relates to: CASSANDRA-11845 Hanging repair in cassandra 2.2.4