[NIFI-9572] Failed to index Provenance Events and (Too many Files)

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.15.2
• Component/s: Core UI
• Assignee: Unassigned
• Reporter: mayki
• Created: 14/Jan/22 09:25
• Updated: 19/Aug/22 15:04

Description

Hello

I have upgraded NIFI 1.15.2 since 2022/01/05

No issue until this night 2022/01/13

• nifi version 1.15.2
• jdk-1.8.0_311

And the limit is high

Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
-bash-4.2@nifi$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) rg
pending signals                 (-i) 63278
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 50000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 10000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
 

 

We got a lot error about provenance_repository, it fill our filesystem logs ..

 

2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
        at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
        at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
        at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
        at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
        at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: Too many open files
 

 

 

We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. But it is impossible to do that if we got this error.

 

Thanks for you help.

 

Regards