[CASSANDRA-10291] Bootstrap hangs on adding new node

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Legacy/Streaming and Messaging

Environment

Debian 7 64 bit
HotSpot JDK 1.7.0_79
Cassandra-2.2.1 via apt-get
1x Intel Quad-Core Xeon E3-1230 / 16GB / 4x1TB SATA / 3x1TB RAID0 data drive

• Severity: Normal
• Since Version: 2.2.0
• Assignee: Unassigned
• Reporter: Ara Sadoyan
• Created: 09/Sep/15 07:44
• Updated: 16/Apr/19 09:30

Description

Adding new node in heavy loaded environment freeze bootstrap. No errors are reported in log files. Some of other other nodes throws "String didn't validate" error, but I;m not sure that this is related.
After restarting node it start bootstrap again and hangs after some time .

nodetool netstats shows :
/data/XXX/XXXX/tmp-la-1184-big-Data.db 5126078789/18345924701 bytes(27%) received from idx:0/192.168.220.16
/data/XXX/XXXX/tmp-la-1233-big-Data.db 7213706459/18600941671 bytes(38%) received from idx:0/192.168.220.22
/data/XXX/XXXX/tmp-la-1599-big-Data.db 8492408759/17572043398 bytes(48%) received from idx:0/192.168.220.12
/data/XXX/XXXX/tmp-la-2066-big-Data.db 15773981555/18508127610 bytes(85%) received from idx:0/192.168.220.18
/data/XXX/XXXX/tmp-la-211-big-Data.db 8274231066/17172754085 bytes(48%) received from idx:0/192.168.220.20

but listing files on local FS shows "No such file or directory"

This happens only if there is significant amount of data. I have 1.5 TB per node on 13 node cluster, we use STCS compaction strategy and flat network topology .