[CXF-6823] Duplicate injectedProviders in ProviderFactory impacting throughput by ~30%

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.5
• Fix Version/s: 3.1.6, 3.0.9, 3.2.0
• Component/s: JAX-RS
• Labels: performance
• Estimated Complexity: Unknown
• Assignee: Sergey Beryozkin
• Reporter: Brendon
• Created: 08/Mar/16 11:11
• Updated: 14/Oct/16 14:13
• Resolved: 08/Mar/16 17:47

Description

We are using CXF to process JAX-RS requests and have noticed a large number of threads blocking in a synchronised block.

ClientProviderFactory and ServerProviderFactory extend ProviderFactory which contains a member variable org.apache.cxf.jaxrs.provider.ProviderFactory#injectedProviders (a LinkedList).

Due to injectProviders being a list, it supports (and contains) duplicates.

In our application, we currently have the following elements in injectProviders:
org.apache.cxf.jaxrs.client.ClientProviderFactory: 26 elements
org.apache.cxf.jaxrs.provider.ServerProviderFactory: 8504 elements

But there are only 4 unique ProviderInfo objects in these lists, namely:
org.apache.cxf.jaxrs.provider.FormEncodingProvider
org.apache.cxf.jaxrs.provider.MultipartProvider
org.apache.cxf.jaxrs.provider.SourceProvider
org.apache.cxf.jaxrs.provider.JAXBElementProvider

The issue is that injectProviders is iterated on each response to clear thread local proxies (org.apache.cxf.jaxrs.provider.ProviderFactory#clearThreadLocalProxies).

Clearing of the thread local proxies relies on a synchronized block to retrieve the thread local proxies from the bus (org.apache.cxf.jaxrs.model.AbstractResourceInfo#getProxyMap).

This means that at scale, the code blocks waiting for entry into this code. I have seen a huge number of threads waiting for entry into this block.

As a test, I have changed injectedProviders to a Set to remove the duplicates and have seen an increase in throughput (RPS) of about ~30% in our application.