[AMBARI-14050] Produce a Warning When Jetty Pool Size Is Too Low And Increase It

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.2.0
• Component/s: ambari-server
• Assignee: Jonathan Hurley
• Reporter: Jonathan Hurley
• Created: 24/Nov/15 20:35
• Updated: 25/Nov/15 02:47
• Resolved: 24/Nov/15 21:38

Description

Ambari's default Jetty thread pool is configured at 25 threads. When deployed on a machine that has 48 "processors", Jetty's calculation for determining how many "Acceptor" and "Selector" threads to create never takes into account the core pool size. As a result, we get:

• 12 "Acceptor" threads (these answer binds to port 8080)
• 12 "Selector" threads (these pick available threads to answer bindings)
• 1 free thread (these are the threads available to do stuff with)

Notice that there's only 1 free thread. This means that Jetty effectively makes Ambari a single-threaded web application on larger systems!

The "fix" is to increase the thread count in the ambari.properties, however nobody knows to do this since no warning is produced.

I suggest:

• Log a warning on server startup if we detect that the core pool size is less than 3/4 of the available processors on the system.

• Automatically increase the core pool size to the min(100, double existing size) when the above is detected

Issue Links

• links to: Reviewboard