Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15208

Listing the same data directory multiple times can result in an java.lang.AssertionError: null on startup

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Triage Needed
    • Priority: Normal
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Local/Config
    • Labels:
      None
    • Platform:
      All
    • Impacts:
      None

      Description

      Listing the same data directory multiple times in the yaml can result in an java.lang.AssertionError: null on startup.

      This error will only happen if Cassandra was stopped part way through an sstable operation (i.e a compaction) and is restarted

      Error: 

      Exception (java.lang.AssertionError) encountered during startup: null
      java.lang.AssertionError
      at org.apache.cassandra.db.lifecycle.LogReplicaSet.addReplica(LogReplicaSet.java:63)
      at java.util.ArrayList.forEach(ArrayList.java:1257)
      at org.apache.cassandra.db.lifecycle.LogReplicaSet.addReplicas(LogReplicaSet.java:57)
      at org.apache.cassandra.db.lifecycle.LogFile.<init>(LogFile.java:147)
      at org.apache.cassandra.db.lifecycle.LogFile.make(LogFile.java:95)
      at org.apache.cassandra.db.lifecycle.LogTransaction$LogFilesByName.removeUnfinishedLeftovers(LogTransaction.java:476)
      at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      at java.util.HashMap$EntrySpliterator.tryAdvance(HashMap.java:1717)
      at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
      at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498)
      at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
      at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
      at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
      at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
      at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454)
      at org.apache.cassandra.db.lifecycle.LogTransaction$LogFilesByName.removeUnfinishedLeftovers(LogTransaction.java:471)
      at org.apache.cassandra.db.lifecycle.LogTransaction.removeUnfinishedLeftovers(LogTransaction.java:438)
      at org.apache.cassandra.db.lifecycle.LogTransaction.removeUnfinishedLeftovers(LogTransaction.java:430)
      at org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:549)
      at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:658)
      at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:275)
      at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620)
      at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732)
      ERROR o.a.c.service.CassandraDaemon Exception encountered during startup
      java.lang.AssertionError: null
      at org.apache.cassandra.db.lifecycle.LogReplicaSet.addReplica(LogReplicaSet.java:63) ~[apache-cassandra-3.11.4.jar:3.11.4]
      at java.util.ArrayList.forEach(ArrayList.java:1257) ~[na:1.8.0_171]
      at org.apache.cassandra.db.lifecycle.LogReplicaSet.addReplicas(LogReplicaSet.java:57) ~[apache-cassandra-3.11.4.jar:3.11.4]
      at org.apache.cassandra.db.lifecycle.LogFile.<init>(LogFile.java:147) ~[apache-cassandra-3.11.4.jar:3.11.4]
      at org.apache.cassandra.db.lifecycle.LogFile.make(LogFile.java:95) ~[apache-cassandra-3.11.4.jar:3.11.4]
      at org.apache.cassandra.db.lifecycle.LogTransaction$LogFilesByName.removeUnfinishedLeftovers(LogTransaction.java:476) ~[apache-cassandra-3.11.4.jar:3.11>
      at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_171]
      at java.util.HashMap$EntrySpliterator.tryAdvance(HashMap.java:1717) ~[na:1.8.0_171]
      at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) ~[na:1.8.0_171]
      at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) ~[na:1.8.0_171]
      at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) ~[na:1.8.0_171]
      at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_171]
      at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) ~[na:1.8.0_171]
      at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) ~[na:1.8.0_171]
      at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_171]
      at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454) ~[na:1.8.0_171]
      at org.apache.cassandra.db.lifecycle.LogTransaction$LogFilesByName.removeUnfinishedLeftovers(LogTransaction.java:471) ~[apache-cassandra-3.11.4.jar:3.11>
      at org.apache.cassandra.db.lifecycle.LogTransaction.removeUnfinishedLeftovers(LogTransaction.java:438) ~[apache-cassandra-3.11.4.jar:3.11.4]
      at org.apache.cassandra.db.lifecycle.LogTransaction.removeUnfinishedLeftovers(LogTransaction.java:430) ~[apache-cassandra-3.11.4.jar:3.11.4]
      at org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:549) ~[apache-cassandra-3.11.4.jar:3.11.4]
      at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:658) ~[apache-cassandra-3.11.4.jar:3.11.4]
      at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:275) [apache-cassandra-3.11.4.jar:3.11.4]
      at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) [apache-cassandra-3.11.4.jar:3.11.4]
      at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) [apache-cassandra-3.11.4.jar:3.11.4]

       Steps to recreate this behaviour:

      • Write data to Cassandra, wait for a largish compaction to start.
      • Kill Cassandra part way through the compaction
      • Check for the existence of a transaction log file in the data directory
      cd /var/lib/cassandra/data
      find . -type f -iname '*.log'
      ./single/table0-3c161330a68d11e9af45010f0154fff5/md_txn_compaction_b492cd30-a68d-11e9-b727-010f0154fff5.log
      • Duplicate the data directory entry in the yaml:
      data_file_directories:
        - /var/lib/cassandra/data
        - /var/lib/cassandra/data
      • Restart Cassandra.

      I have tested that this behaviour exists in the following versions of Cassandra:

      • 3.11.4
      • 3.0.15

      Workaround:

      • Remove the duplicate data directory entry from the Cassandra yaml

      Possible solutions:

      • Validate and filter data_file_directories input to ensure each data directory is unique
      • Fail gracefully, with a more meaningful error message.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Damien Stevenson Damien Stevenson
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: