Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-10794

inflight hashmap of DefaultInflightRepository is slowly increasing to cause memory leak

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Unknown

    Description

      We're using camel 2.15.2 to do url routing, for different url mapping to backend endpoints which are all rest service based.

      The workflow looks like client directly hit our camel application, then based on predefined url path we add customized processor to request headers, then route to backend service. Until get the response from downstream service, again add customized headers to response then return to client side.

      So far total daily request is more than 20 million per server, we found out available memory is getting lower and lower to indicate a slow memory leak.

      After analyze the java heap dump on that server, it's clearly saying the instance of "org.apache.camel.impl.DefaultInflightRepository" is using most heap.

      Inside DefaultInflightRepository instance, there's a ConcurrentMap "inflight" which actually hold most data, the pattern is per daily hit 20 million requests we'll have incremental 400-500 size to ConcurrentMap.

      Please refer to sample data from heap dump as below.

      key: "ID-hostname-0-125588844"
      Value:"org.apache.camel.impl.DefaultExchange @ 0x76ba87bc0"

      From camel message history, we can see that our camel application already send the request to downstream service that is working fine.
      While the callback response never get called, so camel not remove the key from ConcurrentMap.

      Can you help to check this issue with camel and is there any issue related to netty4-http component?

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            luhaobuick Hao Lu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment