Uploaded image for project: 'WSS4J'
  1. WSS4J
  2. WSS-431

Performance bottleneck in MemoryReplayCache on high load

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.9
    • Fix Version/s: 1.6.10
    • Component/s: WSS4J Core
    • Labels:
      None

      Description

      I've been testing some WS-Security policy enabled endpoints running on Apache CXF 2.6.6 + WSS4J 1.6.9. In the testing scenario I have hundreds of ws clients hitting the same endpoint; I'm seeing most of the server treads blocked as follows:

      "http-/172.19.1.6:8080-41" daemon prio=10 tid=0x00007f6bb41c2800 nid=0x6599 waiting for monitor entry [0x00007f6b6e0ac000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at org.apache.ws.security.cache.MemoryReplayCache.processTokenExpiry(MemoryReplayCache.java:89)
      	- waiting to lock <0x00000000b0a71170> (a java.util.Collections$SynchronizedSet)
      	at org.apache.ws.security.cache.MemoryReplayCache.contains(MemoryReplayCache.java:77)
      	at org.apache.ws.security.processor.SignatureProcessor.testMessageReplay(SignatureProcessor.java:674)
      	at org.apache.ws.security.processor.SignatureProcessor.verifyXMLSignature(SignatureProcessor.java:416)
      	at org.apache.ws.security.processor.SignatureProcessor.handleToken(SignatureProcessor.java:231)
      	at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:396)
      	at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:277)
      	at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:96)
      	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
      	- locked <0x00000000ede3b8f8> (a org.apache.cxf.phase.PhaseInterceptorChain)
      	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
      	at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:237)
      [...]
      

      while one of them does:

      "http-/172.19.1.6:8080-17" daemon prio=10 tid=0x00007f6bb418d000 nid=0x6581 runnable [0x00007f6b6fdfb000]
         java.lang.Thread.State: RUNNABLE
      	at org.apache.ws.security.cache.MemoryReplayCache.processTokenExpiry(MemoryReplayCache.java:92)
      	- locked <0x00000000b0a71170> (a java.util.Collections$SynchronizedSet)
      	at org.apache.ws.security.cache.MemoryReplayCache.contains(MemoryReplayCache.java:77)
      	at org.apache.ws.security.processor.SignatureProcessor.testMessageReplay(SignatureProcessor.java:674)
      	at org.apache.ws.security.processor.SignatureProcessor.verifyXMLSignature(SignatureProcessor.java:416)
      	at org.apache.ws.security.processor.SignatureProcessor.handleToken(SignatureProcessor.java:231)
      	at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:396)
      	at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:277)
      	at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:96)
      	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
      	- locked <0x00000000ebba7660> (a org.apache.cxf.phase.PhaseInterceptorChain)
      	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
      	at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:237)
      [...]
      

      So basically the iteration over the cache HashSet in the synchronized block is killing performances.

      I'm aware the MemoryReplayCache is just a simple impl of ReplayCache to be used when EHCache is not available, however I have a proposal for a fix in it that would solve the problem.
      Using a sorted collection we can trade a bit of the cache insertion time to speed up the eviction (processTokenExpiry) process.

      1. patch-cache.diff
        5 kB
        Alessio Soldano

        Activity

        Hide
        asoldano Alessio Soldano added a comment -

        Attaching a patch for using a Date key based sorted map as cache, supported by a synchronized hashset for fast lookup of identifiers.

        Show
        asoldano Alessio Soldano added a comment - Attaching a patch for using a Date key based sorted map as cache, supported by a synchronized hashset for fast lookup of identifiers.

          People

          • Assignee:
            coheigea Colm O hEigeartaigh
            Reporter:
            asoldano Alessio Soldano
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development