[CASSANDRA-12435] Cassandra timeout when using secondary index in cluster mode.

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 3.11.5
• Component/s: Feature/2i Index

Environment

C* 3.5 | java-driver|AWS g2.2xlarge|ubuntu 14.04

• Severity: Normal
• Assignee: Unassigned
• Reporter: Shijian Liu
• Created: 11/Aug/16 07:43
• Updated: 01/Aug/21 11:28
• Resolved: 09/Aug/17 04:22

Description

I have a 3-node cluster with a keyspace with RF=3, then I created a table like below and inserted some data(about 10MB).

CREATE TABLE vehicle_fuzzy_plate (
    day int,
    hour int,
    repo int,
    sensor_id int,
    ts timestamp,
    id timeuuid,
    plate_text_pattern_70 text,
    plate_text_pattern_10 text,
    plate_text_pattern_11 text,
    plate_text_pattern_12 text,
    plate_text_pattern_13 text,
    plate_text_pattern_14 text,
    plate_text_pattern_15 text,
    plate_text_pattern_16 text,
    plate_text_pattern_20 text,
    plate_text_pattern_21 text,
    plate_text_pattern_22 text,
    plate_text_pattern_23 text,
    plate_text_pattern_24 text,
    plate_text_pattern_25 text,
    plate_text_pattern_30 text,
    plate_text_pattern_31 text,
    plate_text_pattern_32 text,
    plate_text_pattern_33 text,
    plate_text_pattern_34 text,
    plate_text_pattern_40 text,
    plate_text_pattern_41 text,
    plate_text_pattern_42 text,
    plate_text_pattern_43 text,
    plate_text_pattern_50 text,
    plate_text_pattern_51 text,
    plate_text_pattern_52 text,
    plate_text_pattern_60 text,
    plate_text_pattern_61 text,
    PRIMARY KEY (day, repo, sensor_id, ts, id)
) WITH CLUSTERING ORDER BY (repo ASC, sensor_id ASC, ts ASC, id ASC);
CREATE INDEX plate_text_pattern_10_idx ON vehicle_fuzzy_plate (plate_text_pattern_10);
CREATE INDEX plate_text_pattern_11_idx ON vehicle_fuzzy_plate (plate_text_pattern_11);
CREATE INDEX plate_text_pattern_12_idx ON vehicle_fuzzy_plate (plate_text_pattern_12);
CREATE INDEX plate_text_pattern_13_idx ON vehicle_fuzzy_plate (plate_text_pattern_13);
CREATE INDEX plate_text_pattern_14_idx ON vehicle_fuzzy_plate (plate_text_pattern_14);
CREATE INDEX plate_text_pattern_15_idx ON vehicle_fuzzy_plate (plate_text_pattern_15);
CREATE INDEX plate_text_pattern_16_idx ON vehicle_fuzzy_plate (plate_text_pattern_16);
CREATE INDEX plate_text_pattern_20_idx ON vehicle_fuzzy_plate (plate_text_pattern_20);
CREATE INDEX plate_text_pattern_21_idx ON vehicle_fuzzy_plate (plate_text_pattern_21);
CREATE INDEX plate_text_pattern_22_idx ON vehicle_fuzzy_plate (plate_text_pattern_22);
CREATE INDEX plate_text_pattern_23_idx ON vehicle_fuzzy_plate (plate_text_pattern_23);
CREATE INDEX plate_text_pattern_24_idx ON vehicle_fuzzy_plate (plate_text_pattern_24);
CREATE INDEX plate_text_pattern_25_idx ON vehicle_fuzzy_plate (plate_text_pattern_25);
CREATE INDEX plate_text_pattern_30_idx ON vehicle_fuzzy_plate (plate_text_pattern_30);
CREATE INDEX plate_text_pattern_31_idx ON vehicle_fuzzy_plate (plate_text_pattern_31);
CREATE INDEX plate_text_pattern_32_idx ON vehicle_fuzzy_plate (plate_text_pattern_32);
CREATE INDEX plate_text_pattern_33_idx ON vehicle_fuzzy_plate (plate_text_pattern_33);
CREATE INDEX plate_text_pattern_34_idx ON vehicle_fuzzy_plate (plate_text_pattern_34);
CREATE INDEX plate_text_pattern_40_idx ON vehicle_fuzzy_plate (plate_text_pattern_40);
CREATE INDEX plate_text_pattern_41_idx ON vehicle_fuzzy_plate (plate_text_pattern_41);
CREATE INDEX plate_text_pattern_42_idx ON vehicle_fuzzy_plate (plate_text_pattern_42);
CREATE INDEX plate_text_pattern_43_idx ON vehicle_fuzzy_plate (plate_text_pattern_43);
CREATE INDEX plate_text_pattern_50_idx ON vehicle_fuzzy_plate (plate_text_pattern_50);
CREATE INDEX plate_text_pattern_51_idx ON vehicle_fuzzy_plate (plate_text_pattern_51);
CREATE INDEX plate_text_pattern_52_idx ON vehicle_fuzzy_plate (plate_text_pattern_52);
CREATE INDEX plate_text_pattern_60_idx ON vehicle_fuzzy_plate (plate_text_pattern_60);
CREATE INDEX plate_text_pattern_61_idx ON vehicle_fuzzy_plate (plate_text_pattern_61);
CREATE INDEX plate_text_pattern_70_idx ON vehicle_fuzzy_plate (plate_text_pattern_70);

After that, I send some queries through Java driver with Consistency=Quorum one by one, for example,

SELECT id,sensor_id,ts FROM vehicle_fuzzy_plate WHERE day IN (16993,16994,16995,16996,16997,16998,16999,17000,17001,17002,17003,17004,17005,17006,17007,17008,17009,17010,17011,17012,17013,17014,17015,17016,17017,17018,17019,17020,17021,17022,17023) AND repo IN (234) AND sensor_id IN (123,16) AND ts>=1468226847000 AND ts<=1470818847000 AND plate_text_pattern_70 IN ('70贵JBP378') ALLOW FILTERING;

The weird part is that some queries will response timeout (about 30%). I checked the debug.log and found these exception,

DEBUG [SharedPool-Worker-1] 2016-08-11 14:09:00,885 ReadCallback.java:126 - Timed out; received 1 of 2 responses (including data)
ERROR [MessagingService-Incoming-/192.168.2.122] 2016-08-11 14:09:05,954 CassandraDaemon.java:217 - Exception in thread Thread[MessagingService-Incoming-/192.168.2.122,5,main]
java.lang.ArrayIndexOutOfBoundsException: 74
        at org.apache.cassandra.db.filter.AbstractClusteringIndexFilter$FilterSerializer.deserialize(AbstractClusteringIndexFilter.java:74) ~[apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.SinglePartitionReadCommand$Deserializer.deserialize(SinglePartitionReadCommand.java:1045) ~[apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:667) ~[apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:595) ~[apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114) ~[apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190) ~[apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) ~[apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[apache-cassandra-3.5.jar:3.5]

I try to clean table and send the query, but timeout comes out occasionally too.
However, queries will work fine if I don't use the secondary index, just like this,

SELECT id,sensor_id,ts FROM vehicle_fuzzy_plate WHERE day IN (16993,16994,16995,16996,16997,16998,16999,17000,17001,17002,17003,17004,17005,17006,17007,17008,17009,17010,17011,17012,17013,17014,17015,17016,17017,17018,17019,17020,17021,17022,17023) AND repo IN (234) AND sensor_id IN (123,16) AND ts>=1468226847000 AND ts<=1470818847000;

Everything works fine on a single node with RF=1 with more than 10GB data.

Issue Links

• duplicates: CASSANDRA-13363 Fix racy read command serialization