Started to notice many open file descriptors (100k+) per node, growing at a rate of about 30 per minute in our cluster. After turning off our JMX exporting server(https://github.com/prometheus/jmx_exporter), which gets queried every 30 seconds, the number of file descriptors remained static.
Digging a bit further I ran a jmx dump tool over all the cassandra metrics and tracked the number of file descriptors after each query, boiling it down to a single metric causing the number of file descriptors to increase:
running a query a few times against this metric shows the file descriptors increasing after each query:
it should be noted that the file descriptor is open on a directory, not an actual file