Noticed on a cluster that runs YCSB non-stop in a loop, using a new table each time and deleting an old one, while ensuring that it keeps at least 10 tables to simulate a "loaded" cluster.
After a few weeks of running, the cluster's file descriptor usage started triggering alerts in the monitoring system. The total data consumption never really changes since we keep at most 10 tables. Todd found that we have thousands of really small log block manager containers.
The workaround at the moment is to simply increase the ulimit for open file descriptors.