Major Description
This seems related to NIFI-1726: we've noticed that the content repository takes up increasingly more space over time, even beyond the configured max usage percentage (see images). After restarting the NiFi cluster we get an immediate drop in disk usage with lots of log statements indicating that expired content is being removed.
Not sure if this is related, but we also often get "Too many open files" during this expiration process after NiFi restart, despite lsof indicating a count far lower than our configured nofile and fs-max.
In the environment indicated by the pictures, nifi.content.repository.archive.max.usage.percentage = 50%. Note that the flow itself only has ~240GB queued across the entire cluster, but each content_repository directory has over 360GB on each worker. Also note the disk usage graph increasing above 50% on each worker, until we finally restart and then the usage drops below 50%.
