[CASSANDRA-11470] dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.6, 3.6
• Labels: dtest
• Severity: Normal
• Assignee: Stefania Alborghetti
• Reporter: Philip Thompson

Authors

Stefania Alborghetti

Reviewers

Marcus Eriksson

• Created: 31/Mar/16 16:59
• Updated: 16/Apr/19 09:30
• Resolved: 11/Apr/16 11:33

Description

base_replica_repair_test has failed on trunk with the following exception in the log of node2:

ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception encountered during startup
java.lang.RuntimeException: Failed to list files in /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
        at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53) ~[main/:na]
        at org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547) ~[main/:na]
        at org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725) ~[main/:na]
        at org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690) ~[main/:na]
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567) ~[main/:na]
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555) ~[main/:na]
        at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383) ~[main/:na]
        at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:320) ~[main/:na]
        at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130) ~[main/:na]
        at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107) ~[main/:na]
        at org.apache.cassandra.cql3.restrictions.StatementRestrictions.<init>(StatementRestrictions.java:139) ~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864) ~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811) ~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799) ~[main/:na]
        at org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505) ~[main/:na]
        at org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242) ~[main/:na]
        at org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286) ~[main/:na]
        at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294) ~[main/:na]
        at org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246) ~[main/:na]
        at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875) ~[main/:na]
        at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867) ~[main/:na]
        at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134) ~[main/:na]
        at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124) ~[main/:na]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229) [main/:na]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562) [main/:na]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [main/:na]
Caused by: java.lang.RuntimeException: Failed to list directory files in /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985, inconsistent disk state for transaction [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
        at org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149) ~[main/:na]
        at org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103) ~[main/:na]
        at org.apache.cassandra.db.lifecycle.LogAwareFileLister$$Lambda$48/35984028.accept(Unknown Source) ~[na:na]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[na:1.8.0_45]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_45]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[na:1.8.0_45]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512) ~[na:1.8.0_45]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502) ~[na:1.8.0_45]
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[na:1.8.0_45]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[na:1.8.0_45]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_45]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[na:1.8.0_45]
        at org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:71) ~[main/:na]
        at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:49) ~[main/:na]
        ... 25 common frames omitted

example failure:

http://cassci.datastax.com/job/trunk_dtest/1092/testReport/materialized_views_test/TestMaterializedViews/base_replica_repair_test

Failed on CassCI build trunk_dtest #1092

I've attached the logs from the failure in build 1092.