[CASSANDRA-13471] [PerDiskMemtableFlushWriter_0:1312] 2017-04-25 09:48:14,818 CassandraDaemon.java:226 - Exception in thread Thread[PerDiskMemtableFlushWriter_0:1312,5,main]

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 3.11.x

Environment

Centos7.3 x86_64, cassandra 3.9

• Severity: Normal
• Assignee: Unassigned
• Reporter: Chernishev Aleksandr
• Created: 25/Apr/17 09:04
• Updated: 16/Apr/19 09:30

Description

with sasi index in table test.object:

 CREATE TABLE test.object (
    bname text,
    name text,
    acl text,
    checksum text,
    chunksize bigint,
    contenttype text,
    creationdate timestamp,
    inode uuid,
    lastmodified timestamp,
    metadata map<text, text>,
    parts map<uuid, bigint>,
    size bigint,
    storageclass text,
    version uuid,
    PRIMARY KEY (bname, name)
) WITH CLUSTERING ORDER BY (name ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '128', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 0.5
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';
CREATE INDEX inode_index ON test.object (inode);

im get error(on random servers in cluster in random times) :

ERROR [PerDiskMemtableFlushWriter_0:1312] 2017-04-25 09:48:14,818 CassandraDaemon.java:226 - Exception in thread Thread[PerDiskMemtableFlushWriter_0:1312,5,main]
java.lang.RuntimeException: Last written key DecoratedKey(d3f60675-e56e-4551-b468-d4e31c8ee82b, d3f60675e56e4551b468d4e31c8ee82b) >= current key DecoratedKey(6a473364-2f43-3876-574d-693461546d73, d3f6355a5bd8415290668f987a15594c) writing into /data3/test/object-f40120c028d111e78e26c57aefc93bac/.inode_index/mc-26-big-Data.db
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:122) ~[apache-cassandra-3.9.0.jar:3.9.0]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:161) ~[apache-cassandra-3.9.0.jar:3.9.0]
        at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48) ~[apache-cassandra-3.9.0.jar:3.9.0]
        at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:458) ~[apache-cassandra-3.9.0.jar:3.9.0]
        at org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:493) ~[apache-cassandra-3.9.0.jar:3.9.0]
        at org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:380) ~[apache-cassandra-3.9.0.jar:3.9.0]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

After that in debug log repeated messages:

 DEBUG [MemtablePostFlush:405] 2017-04-25 09:48:15,944 ColumnFamilyStore.java:936 - forceFlush requested but everything is clean in object

Commitlog not truncate and grows.