[SMXCOMP-504] ConcurrentModificationExceptions under heavy load in CXF BC

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: servicemix-cxf-bc-2008.01, servicemix-cxf-se-2008.01
• Fix Version/s: 3.2.4, servicemix-cxf-bc-2009.02, servicemix-cxf-se-2009.02
• Component/s: servicemix-cxf-bc, servicemix-cxf-se
• Assignee: Freeman Yue Fang
• Reporter: Martin Murphy
• Created: 10/Apr/09 16:24
• Updated: 29/Apr/09 09:12
• Resolved: 29/Apr/09 09:12

Description

Under heavy load ServiceMix throws occasional ConcurrentModificationExceptions with this stack trace:

Caused by: java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at org.apache.cxf.phase.PhaseInterceptorChain.add(PhaseInterceptorChain.java:168)
at org.apache.cxf.phase.PhaseInterceptorChain.add(PhaseInterceptorChain.java:160)
at org.apache.servicemix.cxfbc.CxfBcProvider.process(CxfBcProvider.java:227)
at org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:621)
at org.apache. servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:575)
at org.apache.servicemix.common.AsyncBaseLifeCycle.onMessageExchange(AsyncBaseLifeCycle.java:531)
at org.apache.servicemix.common.SyncLifeCycleWrapper.onMessageExchange(SyncLifeCycleWrapper.java:60)
at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:623)
at org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:16

The reason for this is that the CXF example from the documentation adds the interceptors as a list :

<cxfbc:inInterceptors>
    <ref bean="LoggingInInterceptor"/>
</cxfbc:inInterceptors>
<cxfbc:outInterceptors>
    <ref bean="LoggingOutInterceptor"/>
</cxfbc:outInterceptors>
<cxfbc:inFaultInterceptors>
    <ref bean="LoggingInInterceptor"/>
</cxfbc:inFaultInterceptors>
<cxfbc:outFaultInterceptors>
    <ref bean="LoggingOutInterceptor"/>
</cxfbc:outFaultInterceptors>

For example this then gets set as a list when injected by Spring

    List<Interceptor> in = new CopyOnWriteArrayList<Interceptor>();
// ... SNIP ...
    /**
        * Specifies a list of interceptors used to process requests recieved
        * by the endpoint.
        *
        * @param interceptors   a list of <code>Interceptor</code> objects
        * @org.apache.xbean.Property description="a list of beans configuring interceptors that process incoming requests"
        * */
    public void setInInterceptors(List<Interceptor> interceptors) {
        in = interceptors;
    }

The CopyOnWriteArrayList is lost and can lead to the ConcurrentModificationException.
One possible solution is to update the Spring configuration for the interceptors to wrap it in a CopyOnWriteArrayList.

<property name="inInterceptors">
    <util:list list-class="java.util.concurrent.CopyOnWriteArrayList">
        <ref bean="LoggingInInterceptor"/>
    </util:list>
</property>
<property name="outInterceptors">
    <util:list list-class="java.util.concurrent.CopyOnWriteArrayList">
        <ref bean="LoggingOutInterceptor"/>
    </util:list>
</property>
<property name="inFaultInterceptors">
    <util:list list-class="java.util.concurrent.CopyOnWriteArrayList">
        <ref bean="LoggingInInterceptor"/>
    </util:list>
</property>
<property name="outFaultInterceptors">
    <util:list list-class="java.util.concurrent.CopyOnWriteArrayList">
        <ref bean="LoggingOutInterceptor"/>
    </util:list>
</property>

This is probably overkill since the setter methods could probably handle
this by adding the list to the existing CopyOnWriteArrayList rather than
overwriting the existing object, for example:

    /**
        * Specifies a list of interceptors used to process requests recieved
        * by the endpoint.
        *
        * @param interceptors   a list of <code>Interceptor</code> objects
        * @org.apache.xbean.Property description="a list of beans configuring interceptors that process incoming requests"
        * */
    public void setInInterceptors(List<Interceptor> interceptors) {
        in.addAll(interceptors);
    }

This would allow current configurations to avoid possible ConcurrentModificationExceptions