[CXF-2992] response context concurrent issue

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.2
• Fix Version/s: 2.2.12, 2.3.1
• Component/s: Core

Environment

Java: Java(TM) 2 Runtime Environment, Standard Edition (IBM build 1.5.0_16-b02
20080907 (SR8 + IZ29767 + IZ30684 + IZ31214 + IZ31213)) Java HotSpot(TM) Server VM
Os: SunOS 5.10 Generic_142900-03 sun4v sparc SUNW,SPARC-Enterprise-T5220

• Assignee: Daniel Kulp
• Reporter: Justinas Dabravolskas
• Created: 14/Sep/10 12:09
• Updated: 13/Sep/11 15:28
• Resolved: 01/Oct/10 17:46

Description

After running an application with CXF for longer time, we have some threads(4-5) that are using about 100% cpu each. It occurred three times already and the only solution to this problem is to restart an application server.

PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
10992 was 97 0.0 0.0 0.0 0.0 2.7 0.0 0.0 3 43 7 1 java/966
10992 was 97 0.0 0.0 0.0 0.0 2.7 0.0 0.0 3 40 7 1 java/491
10992 was 97 0.0 0.0 0.0 0.0 2.7 0.0 0.0 3 60 7 1 java/1251
10992 was 97 0.0 0.0 0.0 0.0 2.7 0.0 0.0 3 102 7 1 java/1280

All hung threads has the same stack trace "default-workqueue-232" prio=3 tid=0x00000001065c2c50 nid=0x4e3 runnable [0xfffffffdf86fe000..0xfffffffdf86ff9a8]
at java.util.HashMap.put(HashMap.java:420)
at java.util.HashMap.putAll(HashMap.java:570)
at org.apache.cxf.endpoint.ClientImpl.processResult(ClientImpl.java:520)
at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:659)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2132)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream$1.run(HTTPConduit.java:2020)
at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$2.run(AutomaticWorkQueueImpl.java:243)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595).

ClientImpl tries to add response message to response context and it enters into an infinitive loop inside HashMap. (line 420 for (Entry<K,V> e = table[i]; e != null; e = e.next) )
Though at first glance response context is access based on the current thread, I think the only possible way HashMap may enter this infinitive loop is concurrent modification somewhere inside CXF code.

Issue Links

• is duplicated by: CXF-3800 ClientImpl reuse ResponseContext cause ConcurrentModificationException