[CASSANDRA-12978] mx4j -> HTTP 500 -> ConcurrentModificationException

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Won't Fix
• Fix Version/s: 2.1.6
• Component/s: Legacy/Tools

Environment

Debian, Single cluster, 2 data centres, E5-2620 v3, 16GB, RAID1 SSD Commit log, RAID10 15k HDD data

• Severity: Critical
• Assignee: Unassigned
• Reporter: Rob Emery
• Created: 30/Nov/16 17:49
• Updated: 16/Apr/19 09:30
• Resolved: 12/Dec/17 16:54

Description

We run some checks from our Monitoring software that rely on mx4j.

The checks typically grab some xml via HTTP request and parse it. For
example, CF Stats on 'MyKeySpace' and 'MyColumnFamily' are retrieved
using:

http://cassandra001:8081/mbean?template=identity&objectname=org.apache.cassandra.db%3Atype%3DColumnFamilies%2Ckeyspace%3DMyKeySpace%2Ccolumnfamily%3DMyColumnFamily

The checks run each minute. Periodically they result in a "HTTP 500 internal server error". The HTML body returned is empty.

Experimentally we ran Cassandra in the foreground on one node and reproduced the problem. this elicited the following stack trace:

javax.management.RuntimeMBeanException: java.util.ConcurrentModificationException
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at mx4j.tools.adaptor.http.MBeanCommandProcessor.createMBeanElement(MBeanCommandProcessor.java:119)
at mx4j.tools.adaptor.http.MBeanCommandProcessor.executeRequest(MBeanCommandProcessor.java:56)
at mx4j.tools.adaptor.http.HttpAdaptor$HttpClient.run(HttpAdaptor.java:980)
Caused by: java.util.ConcurrentModificationException
at java.util.TreeMap$NavigableSubMap$SubMapIterator.nextEntry(TreeMap.java:1594)
at java.util.TreeMap$NavigableSubMap$SubMapEntryIterator.next(TreeMap.java:1642)
at java.util.TreeMap$NavigableSubMap$SubMapEntryIterator.next(TreeMap.java:1636)
at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
at org.apache.cassandra.utils.StreamingHistogram.sum(StreamingHistogram.java:160)
at org.apache.cassandra.io.sstable.metadata.StatsMetadata.getDroppableTombstonesBefore(StatsMetadata.java:113)
at org.apache.cassandra.io.sstable.SSTableReader.getDroppableTombstonesBefore(SSTableReader.java:2004)
at org.apache.cassandra.db.DataTracker.getDroppableTombstoneRatio(DataTracker.java:507)
at org.apache.cassandra.db.ColumnFamilyStore.getDroppableTombstoneRatio(ColumnFamilyStore.java:3089)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
... 4 more