Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
1.2.0, 1.3.0, 1.4.0, 1.5.0
-
None
-
CENTOS 7
Description
When you have multiple HandleHTTPRequest processors trying to listen on the same port, for every Listen attempt NiFi builds a new thread and never recycles the old thread which eventually leads to NiFi shutting down when reaching the OS limit of the number of threads (default is 10.000).
The following error can be seen in nifi-app.log:
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
This has happened before with version 1.2 and probably even with older versions. but I could also replicate the issue with the latest 1.5 version.
Steps to replicate the issue:
1) build a simple flow with 2 HandleHTTPRequest processors listening on the same port.
2) Start the processors.
— The second HandleHTTPRequest processor starts logging following as expected:
2018-02-02 16:18:29,518 ERROR [Timer-Driven Process Thread-3] o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=af013c62-b26f-1eeb-ae81-8423c70bdc7f] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server: {}
org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server
....
Caused by: java.net.BindException: Address already in use
...
... 12 common frames omitted
3) Go to the Summary section in NiFi and watch the number of threads going up to 9959.
With above, I had processors scheduled on primary node only as to not affect every node.
If you stop the second HandleHTTPRequest processor the threads stop climbing, but are not released.
After this, NiFi will soon stop.
A restart of NIFi is required to release these threads.
Attachments
Attachments
Issue Links
- links to
