[CASSANDRA-18770] Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5

• Type: Bug
• Status: Triage Needed
• Priority: Normal
• Resolution: Unresolved
• Platform: Java8
• Assignee: Unassigned
• Reporter: vaibhav khedkar
• Created: 16/Aug/23 17:19
• Updated: 17/Aug/23 11:50

Description

We recently upgraded our fleet of ~2500 Cassandra instances from 3.11.9 to 4.0.5.

After the upgrade, we are seeing a unique issue where the compacted SSTables's file descriptors are still present and are never cleared. This is causing false disk alerts. We have to restart nodes very often in order to reclaim the disk space.

We use the Apache version of C* along with CentOS Linux 7.9.2009 and Java 8.

We are seeing it for all the tables (Filter, Data ..etc ) when checked with `lsof` command

<path>/nb-1-big-Statistics.db.tmp (deleted)
<path>/nb-3-big-Statistics.db.tmp (deleted)
<path>/nb-2-big-Data.db (deleted)
<path>/nb-2-big-Statistics.db.tmp (deleted)
<path>/nb-2-big-Index.db (deleted)
<path>/nb-2-big-Statistics.db.tmp (deleted)
<path>/nb-2-big-CompressionInfo.db (deleted)
<path>/nb-2-big-Filter.db (deleted)
<path>/nb-2-big-Summary.db (deleted)
<path>/nb-2-big-Digest.crc32 (deleted)