ActiveMQ
  1. ActiveMQ
  2. AMQ-3276

ConcurrentModificationException in embedded 5.5.0 broker

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.5.0
    • Fix Version/s: 5.6.0
    • Component/s: Broker
    • Labels:
      None
    • Environment:

      jdk 1.6.0_24, Spring 3.0.5, ActiveMQ 5.5.0, Camel 2.7.0, slf4j 1.6.1

      Description

      We just migrated from ActiveMQ 5.4.2 to ActiveMQ 5.5.0. So far so good, with one exception (pun not intended). In one case where we have an embedded broker, we're seeing this exception get logged on occasion:

      WARNING; 08-Apr-2011 11:11:41; tid:45931; TransportConnection stopAsync; cannot create async transport stopper thread.. not waiting for stop to complete, reason:
      java.util.ConcurrentModificationException
      at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
      at java.util.HashMap$EntryIterator.next(HashMap.java:834)
      at java.util.HashMap$EntryIterator.next(HashMap.java:832)
      at java.util.HashMap.putAllForCreate(HashMap.java:435)
      at java.util.HashMap.<init>(HashMap.java:225)
      at org.slf4j.helpers.BasicMDCAdapter.getCopyOfContextMap(BasicMDCAdapter.java:130)
      at org.slf4j.MDC.getCopyOfContextMap(MDC.java:182)
      at org.apache.activemq.util.MDCHelper.getCopyOfContextMap(MDCHelper.java:30)
      at org.apache.activemq.broker.TransportConnection.stopAsync(TransportConnection.java:946)
      at org.apache.activemq.broker.TransportConnection.processShutdown(TransportConnection.java:353)
      at org.apache.activemq.command.ShutdownInfo.visit(ShutdownInfo.java:35)
      at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306)
      at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
      at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
      at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
      at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:218)
      at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:127)
      at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)

      Up until this morning, I had only seen that logged during shutdown of the app context. But just now, it popped out independently, out of the blue.

      For what it's worth, here's the app context config:

      <broker xmlns="http://activemq.apache.org/schema/core"
      id="embeddedActivemqBroker"
      useJmx="true"
      persistent="true"
      schedulerSupport="false"
      advisorySupport="false"
      enableStatistics="true">
      <destinationPolicy>
      <policyMap>
      <policyEntries>
      <policyEntry queue=">" producerFlowControl="false" memoryLimit="20mb"/>
      </policyEntries>
      </policyMap>
      </destinationPolicy>
      <persistenceAdapter>
      <kahaDB directory="$

      {EmbeddedBroker.dataDirectory}

      "
      concurrentStoreAndDispatchQueues="false"/>
      </persistenceAdapter>
      </broker>

      Has anybody else seen this ConcurrentModificationException happening with 5.5.0 (or otherwise)? Any ideas?

      I suppose I should also mention that we're using slf4j 1.6.1. Not sure if that has anything to do with this, since the stack trace does show it happening in slf4j land...

      <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
      <version>1.6.1</version>
      <scope>runtime</scope>
      </dependency>
      <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-jdk14</artifactId>
      <version>1.6.1</version>
      <scope>runtime</scope>
      </dependency>

        Activity

        Show
        Hondong added a comment - You can download it here: http://repo.fusesource.com/nexus/content/repositories/releases/org/apache/activemq/apache-activemq/5.5.0-fuse-00-43/
        Hide
        Hondong added a comment -

        Hi, have a look at Fuse 5.5.0 00.43: http://fusesource.com/wiki/display/ProdInfo/FUSE+Message+Broker+v5.5.0-fuse+Release+Notes
        This Issue seems to be fixed there.

        Show
        Hondong added a comment - Hi, have a look at Fuse 5.5.0 00.43: http://fusesource.com/wiki/display/ProdInfo/FUSE+Message+Broker+v5.5.0-fuse+Release+Notes This Issue seems to be fixed there.
        Hide
        Falko Modler added a comment -

        Dan Checkoway wrote:
        "I'm guessing 5.6.0 will be a while? Do you think there will be a 5.5.1 release in the near term?"

        Same questions here!

        Show
        Falko Modler added a comment - Dan Checkoway wrote: "I'm guessing 5.6.0 will be a while? Do you think there will be a 5.5.1 release in the near term?" Same questions here!
        Hide
        Daniel Kulp added a comment -


        Just a note that while testing CXF with 5.5.0, I also run into this occasionally. The CXF test suite will need to remain on 5.4.2 for now.

        Show
        Daniel Kulp added a comment - Just a note that while testing CXF with 5.5.0, I also run into this occasionally. The CXF test suite will need to remain on 5.4.2 for now.
        Hide
        Dan Checkoway added a comment -

        Thanks for the very quick turnaround, Gary. Unfortnately, the only place we've seen this issue is in our loaded production environment. SNAPSHOTs are all but off limits, sad to say.

        I'm guessing 5.6.0 will be a while? Do you think there will be a 5.5.1 release in the near term?

        Show
        Dan Checkoway added a comment - Thanks for the very quick turnaround, Gary. Unfortnately, the only place we've seen this issue is in our loaded production environment. SNAPSHOTs are all but off limits, sad to say. I'm guessing 5.6.0 will be a while? Do you think there will be a 5.5.1 release in the near term?
        Hide
        Gary Tully added a comment -

        reverted the explicit copy/set of the MDC context on thread creation as the underlying mdc impl used an inherited thread local for the context which makes the copy unnecessary and can cause contention.
        the changes in http://svn.apache.org/viewvc?view=revision&revision=1090320 should resolve the concurrent mod exception. I kicked off a new 5.6-SNAPSHOT deploy so if you get a chance please validate.

        Show
        Gary Tully added a comment - reverted the explicit copy/set of the MDC context on thread creation as the underlying mdc impl used an inherited thread local for the context which makes the copy unnecessary and can cause contention. the changes in http://svn.apache.org/viewvc?view=revision&revision=1090320 should resolve the concurrent mod exception. I kicked off a new 5.6-SNAPSHOT deploy so if you get a chance please validate.

          People

          • Assignee:
            Gary Tully
            Reporter:
            Dan Checkoway
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development