[CASSANDRA-4571] Strange permament socket descriptors increasing leads to "Too many open files"

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 1.1.5

Environment

CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.

java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)

• Severity: Critical
• Assignee: Jonathan Ellis
• Reporter: Serg Shnerson

Authors

Jonathan Ellis

Reviewers

Sam Tunnicliffe

• Created: 23/Aug/12 17:20
• Updated: 16/Apr/19 09:32
• Resolved: 04/Sep/12 19:09

Description

On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"

java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
" And number of this rows constantly increasing. After about 24 hours this situation leads to error.
We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).