[CASSANDRA-14641] Nodetool toppartitions raises java.lang.IndexOutOfBoundsException

• Type: Bug
• Status: Open
• Priority: Low
• Resolution: Unresolved
• Component/s: Tool/nodetool

Environment

Linux 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

• Severity: Low
• Assignee: Unassigned
• Reporter: Irina Truong
• Created: 13/Aug/18 23:38
• Updated: 16/Apr/19 09:29

Description

I get this exception in most cases when using nodetool.toppartitions command:

irina@host:/usr/local/cassandra/bin$ ./nodetool toppartitions casterisk events_2018_08_13 1000
error: index (4) must be less than size (4)
-- StackTrace --
java.lang.IndexOutOfBoundsException: index (4) must be less than size (4)
	at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310)
	at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:292)
	at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:65)
	at org.apache.cassandra.db.marshal.CompositeType.getAndAppendComparator(CompositeType.java:148)
	at org.apache.cassandra.db.marshal.AbstractCompositeType.getString(AbstractCompositeType.java:174)
	at org.apache.cassandra.db.ColumnFamilyStore.finishLocalSampling(ColumnFamilyStore.java:1588)
	at sun.reflect.GeneratedMethodAccessor1899.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
	at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:324)
	at sun.rmi.transport.Transport$1.run(Transport.java:200)
	at sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Occasionally, nodetool will return a proper response. But in most cases, no.

Cassandra version 3.0.8.

table schema:

cqlsh:casterisk> describe table events_2018_08_14;

CREATE TABLE casterisk.events_2018_08_14 (
    event_type text,
    apikey text,
    url text,
    period timestamp,
    ts timestamp,
    event_id blob,
    d blob,
    PRIMARY KEY ((event_type, apikey, url, period), ts, event_id)
) WITH COMPACT STORAGE
    AND CLUSTERING ORDER BY (ts ASC, event_id ASC)
    AND bloom_filter_fp_chance = 0.1
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

cluster info:

$ ./nodetool describecluster
Cluster Information:
        Name: cass-prod2
        Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
        Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
        Schema versions:
                55480b40-8187-320d-9bbb-f9ddad3b94ba: [10.22.0.214, 10.22.0.20, 10.22.0.210, 10.22.0.147, 10.22.0.241, 10.22.0.81, 10.22.0.158, 10.22.0.90, 10.22.0.27, 10.22.0.184, 10.22.0.130, 10.22.0.47, 10.22.0.13, 10.22.0.234, 10.22.0.74]