Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-12978

mx4j -> HTTP 500 -> ConcurrentModificationException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Urgent
    • Resolution: Won't Fix
    • 2.1.6
    • Legacy/Tools
    • None
    • Debian, Single cluster, 2 data centres, E5-2620 v3, 16GB, RAID1 SSD Commit log, RAID10 15k HDD data

    • Critical

    Description

      We run some checks from our Monitoring software that rely on mx4j.

      The checks typically grab some xml via HTTP request and parse it. For
      example, CF Stats on 'MyKeySpace' and 'MyColumnFamily' are retrieved
      using:

      http://cassandra001:8081/mbean?template=identity&objectname=org.apache.cassandra.db%3Atype%3DColumnFamilies%2Ckeyspace%3DMyKeySpace%2Ccolumnfamily%3DMyColumnFamily

      The checks run each minute. Periodically they result in a "HTTP 500 internal server error". The HTML body returned is empty.

      Experimentally we ran Cassandra in the foreground on one node and reproduced the problem. this elicited the following stack trace:

      javax.management.RuntimeMBeanException: java.util.ConcurrentModificationException
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
      at mx4j.tools.adaptor.http.MBeanCommandProcessor.createMBeanElement(MBeanCommandProcessor.java:119)
      at mx4j.tools.adaptor.http.MBeanCommandProcessor.executeRequest(MBeanCommandProcessor.java:56)
      at mx4j.tools.adaptor.http.HttpAdaptor$HttpClient.run(HttpAdaptor.java:980)
      Caused by: java.util.ConcurrentModificationException
      at java.util.TreeMap$NavigableSubMap$SubMapIterator.nextEntry(TreeMap.java:1594)
      at java.util.TreeMap$NavigableSubMap$SubMapEntryIterator.next(TreeMap.java:1642)
      at java.util.TreeMap$NavigableSubMap$SubMapEntryIterator.next(TreeMap.java:1636)
      at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
      at org.apache.cassandra.utils.StreamingHistogram.sum(StreamingHistogram.java:160)
      at org.apache.cassandra.io.sstable.metadata.StatsMetadata.getDroppableTombstonesBefore(StatsMetadata.java:113)
      at org.apache.cassandra.io.sstable.SSTableReader.getDroppableTombstonesBefore(SSTableReader.java:2004)
      at org.apache.cassandra.db.DataTracker.getDroppableTombstoneRatio(DataTracker.java:507)
      at org.apache.cassandra.db.ColumnFamilyStore.getDroppableTombstoneRatio(ColumnFamilyStore.java:3089)
      at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
      at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
      at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
      at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
      at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
      ... 4 more

      Attachments

        Activity

          People

            Unassigned Unassigned
            re_weavers Rob Emery
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: