[WSS-431] Performance bottleneck in MemoryReplayCache on high load

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.9
• Fix Version/s: 1.6.10
• Component/s: WSS4J Core
• Assignee: Colm O hEigeartaigh
• Reporter: Alessio Soldano
• Created: 12/Mar/13 08:32
• Updated: 25/Mar/13 11:39
• Resolved: 12/Mar/13 14:54

Description

I've been testing some WS-Security policy enabled endpoints running on Apache CXF 2.6.6 + WSS4J 1.6.9. In the testing scenario I have hundreds of ws clients hitting the same endpoint; I'm seeing most of the server treads blocked as follows:

"http-/172.19.1.6:8080-41" daemon prio=10 tid=0x00007f6bb41c2800 nid=0x6599 waiting for monitor entry [0x00007f6b6e0ac000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.ws.security.cache.MemoryReplayCache.processTokenExpiry(MemoryReplayCache.java:89)
	- waiting to lock <0x00000000b0a71170> (a java.util.Collections$SynchronizedSet)
	at org.apache.ws.security.cache.MemoryReplayCache.contains(MemoryReplayCache.java:77)
	at org.apache.ws.security.processor.SignatureProcessor.testMessageReplay(SignatureProcessor.java:674)
	at org.apache.ws.security.processor.SignatureProcessor.verifyXMLSignature(SignatureProcessor.java:416)
	at org.apache.ws.security.processor.SignatureProcessor.handleToken(SignatureProcessor.java:231)
	at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:396)
	at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:277)
	at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:96)
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
	- locked <0x00000000ede3b8f8> (a org.apache.cxf.phase.PhaseInterceptorChain)
	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
	at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:237)
[...]

while one of them does:

"http-/172.19.1.6:8080-17" daemon prio=10 tid=0x00007f6bb418d000 nid=0x6581 runnable [0x00007f6b6fdfb000]
   java.lang.Thread.State: RUNNABLE
	at org.apache.ws.security.cache.MemoryReplayCache.processTokenExpiry(MemoryReplayCache.java:92)
	- locked <0x00000000b0a71170> (a java.util.Collections$SynchronizedSet)
	at org.apache.ws.security.cache.MemoryReplayCache.contains(MemoryReplayCache.java:77)
	at org.apache.ws.security.processor.SignatureProcessor.testMessageReplay(SignatureProcessor.java:674)
	at org.apache.ws.security.processor.SignatureProcessor.verifyXMLSignature(SignatureProcessor.java:416)
	at org.apache.ws.security.processor.SignatureProcessor.handleToken(SignatureProcessor.java:231)
	at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:396)
	at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:277)
	at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:96)
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
	- locked <0x00000000ebba7660> (a org.apache.cxf.phase.PhaseInterceptorChain)
	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
	at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:237)
[...]

So basically the iteration over the cache HashSet in the synchronized block is killing performances.

I'm aware the MemoryReplayCache is just a simple impl of ReplayCache to be used when EHCache is not available, however I have a proposal for a fix in it that would solve the problem.
Using a sorted collection we can trade a bit of the cache insertion time to speed up the eviction (processTokenExpiry) process.