[CASSANDRA-8530] Query on a secondary index creates huge CPU spike + unable to trace

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Component/s: Feature/2i Index, Legacy/CQL

Environment

CentOs 6.5 / Cassandra 2.1.2

• Severity: Normal
• Assignee: Unassigned
• Reporter: Pavel Baranov
• Created: 21/Dec/14 18:33
• Updated: 16/Apr/19 09:31
• Resolved: 11/Nov/15 17:26

Description

After upgrading cassandra from 2.0.10 to 2.1.2 we are having all kinds of issues, especially with performance.

java version "1.7.0_65"

Table creation:

tweets> desc table tweets;

CREATE TABLE tweets.tweets (
    uname text,
    tweet_id bigint,
    tweet text,
    tweet_date timestamp,
    tweet_date_only text,
    uid bigint,
    PRIMARY KEY (uname, tweet_id)
) WITH CLUSTERING ORDER BY (tweet_id ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
    AND comment = ''
    AND compaction = {'min_threshold': '10', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND dclocal_read_repair_chance = 0.0
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.1
    AND speculative_retry = '99.0PERCENTILE';
CREATE INDEX tweets_tweet_date_only_idx ON tweets.tweets (tweet_date_only);
CREATE INDEX tweets_uid ON tweets.tweets (uid);

With Cassandra 2.0.10 this query:

select uname from tweets where uid = 636732672 limit 1;

did not have any issues. After upgrade, I can see the cpu spikes and load avg goes from ~1 to ~13, especially if I execute the query over and over again.

Doing "tracing on" does not work and just returns:
"Statement trace did not complete within 10 seconds"

I've done:
nodetool upgradesstables
recreated indexes

Issue Links

• duplicates: CASSANDRA-10050 Secondary Index Performance Dependent on TokenRange Searched in Analytics