Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Cannot Reproduce
-
0.5.0
-
None
-
None
-
OS: Amazon Linux 2 (4.14.77-86.82.amzn2.x86_64)
Java: 1.8.0_191-b12
ulimits:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31769
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 50000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 10000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
OS: Amazon Linux 2 (4.14.77-86.82.amzn2.x86_64) Java: 1.8.0_191-b12 ulimits: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 31769 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 50000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 10000 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Description
After several days of running C2 the errors are being logged about having "Too Many Open Files". At this point, C2 is not responsive to clients trying to check for new configurations.
33e79e: Too many open files
Apr 29 14:33:22 ********ec2.internal c2.sh[20851]: Caused by: java.nio.file.FileSystemException: /opt/minifi-c2/./files/5233730D-AC58-4853-9C0E-273D0E
Running lsof shows an increasingly larger number of open files. Our MiNiFi instances are checking for new files every few seconds (probably too short), but it seems like maybe some file handles are not being cleaned up.
Restarting C2 clears up this condition and C2 becomes responsive again.