Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-12978

mx4j -> HTTP 500 -> ConcurrentModificationException

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Urgent
    • Resolution: Won't Fix
    • Fix Version/s: 2.1.6
    • Component/s: Legacy/Tools
    • Labels:
      None
    • Environment:

      Debian, Single cluster, 2 data centres, E5-2620 v3, 16GB, RAID1 SSD Commit log, RAID10 15k HDD data

    • Severity:
      Critical

      Description

      We run some checks from our Monitoring software that rely on mx4j.

      The checks typically grab some xml via HTTP request and parse it. For
      example, CF Stats on 'MyKeySpace' and 'MyColumnFamily' are retrieved
      using:

      http://cassandra001:8081/mbean?template=identity&objectname=org.apache.cassandra.db%3Atype%3DColumnFamilies%2Ckeyspace%3DMyKeySpace%2Ccolumnfamily%3DMyColumnFamily

      The checks run each minute. Periodically they result in a "HTTP 500 internal server error". The HTML body returned is empty.

      Experimentally we ran Cassandra in the foreground on one node and reproduced the problem. this elicited the following stack trace:

      javax.management.RuntimeMBeanException: java.util.ConcurrentModificationException
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
      at mx4j.tools.adaptor.http.MBeanCommandProcessor.createMBeanElement(MBeanCommandProcessor.java:119)
      at mx4j.tools.adaptor.http.MBeanCommandProcessor.executeRequest(MBeanCommandProcessor.java:56)
      at mx4j.tools.adaptor.http.HttpAdaptor$HttpClient.run(HttpAdaptor.java:980)
      Caused by: java.util.ConcurrentModificationException
      at java.util.TreeMap$NavigableSubMap$SubMapIterator.nextEntry(TreeMap.java:1594)
      at java.util.TreeMap$NavigableSubMap$SubMapEntryIterator.next(TreeMap.java:1642)
      at java.util.TreeMap$NavigableSubMap$SubMapEntryIterator.next(TreeMap.java:1636)
      at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
      at org.apache.cassandra.utils.StreamingHistogram.sum(StreamingHistogram.java:160)
      at org.apache.cassandra.io.sstable.metadata.StatsMetadata.getDroppableTombstonesBefore(StatsMetadata.java:113)
      at org.apache.cassandra.io.sstable.SSTableReader.getDroppableTombstonesBefore(SSTableReader.java:2004)
      at org.apache.cassandra.db.DataTracker.getDroppableTombstoneRatio(DataTracker.java:507)
      at org.apache.cassandra.db.ColumnFamilyStore.getDroppableTombstoneRatio(ColumnFamilyStore.java:3089)
      at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
      at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
      at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
      at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
      at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
      ... 4 more

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              re_weavers Rob Emery
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: