[NIFI-3978] Site-to-Site HTTP client tests fails on machines with too many cores

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Assignee: Joey Frazee
• Reporter: Joey Frazee
• Created: 25/May/17 15:24
• Updated: 14/Dec/17 07:58
• Resolved: 14/Dec/17 07:58

Description

The S2S http client tests will fail on machines with many cores because the thread pool limits are too low (see error below).

Looks like the reason for this is that Jetty will use acceptors + selectors + request=1 threads, but calculates both the number of acceptors and selectors based on the number of hyper threads available:

• The default number of acceptor tasks is the minimum of 1 and the number of available CPUs divided by 8
• The default number of selectors is equal to half of the number of processors available to the JVM

So suppose you have a 12 core 2 HT machine. Then you'll use 12 * 2 / 8 + 12 * 2 / 2 + 1 = 16 threads and all is well with the current limit of 20. But if you have a 16 core machine, you'll use 16 * 2 / 8 + 16 * 2 / 2 + 1 = 21 and go over the limit.

We should bump this up to 50 which should accommodate builds on 2 x 16 core.

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec <<< FAILURE! - in org.apache.nifi.remote.client.http.TestHttpClient
org.apache.nifi.remote.client.http.TestHttpClient  Time elapsed: 0 sec  <<< ERROR!
java.lang.IllegalStateException: Insufficient threads: max=20 < needed(acceptors=4 + selectors=16 + request=1)
	at org.eclipse.jetty.server.Server.doStart(Server.java:414)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.apache.nifi.remote.client.http.TestHttpClient.setup(TestHttpClient.java:501)

Issue Links

• links to: GitHub Pull Request #1858