[CAMEL-10794] inflight hashmap of DefaultInflightRepository is slowly increasing to cause memory leak

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.15.2
• Component/s: camel-core, camel-guice, camel-netty4-http
• Estimated Complexity: Unknown
• Assignee: Unassigned
• Reporter: Hao Lu
• Created: 07/Feb/17 00:48
• Updated: 07/Feb/17 08:10
• Resolved: 07/Feb/17 08:10

Description

We're using camel 2.15.2 to do url routing, for different url mapping to backend endpoints which are all rest service based.

The workflow looks like client directly hit our camel application, then based on predefined url path we add customized processor to request headers, then route to backend service. Until get the response from downstream service, again add customized headers to response then return to client side.

So far total daily request is more than 20 million per server, we found out available memory is getting lower and lower to indicate a slow memory leak.

After analyze the java heap dump on that server, it's clearly saying the instance of "org.apache.camel.impl.DefaultInflightRepository" is using most heap.

Inside DefaultInflightRepository instance, there's a ConcurrentMap "inflight" which actually hold most data, the pattern is per daily hit 20 million requests we'll have incremental 400-500 size to ConcurrentMap.

Please refer to sample data from heap dump as below.

key: "ID-hostname-0-125588844"
Value:"org.apache.camel.impl.DefaultExchange @ 0x76ba87bc0"

From camel message history, we can see that our camel application already send the request to downstream service that is working fine.
While the callback response never get called, so camel not remove the key from ConcurrentMap.

Can you help to check this issue with camel and is there any issue related to netty4-http component?