Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-6531

Failure to start after unclean shutdown - java.lang.IllegalArgumentException: bufferSize must be positive

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 1.2.14, 2.0.5
    • None
    • Normal

    Description

      We had a severe power outage in the lab that resulted in unclean shutdown of the Cassandra servers. After the power was back I tried to start the cluster. Two out of 6 nodes cannot start because of this exception:

       INFO 20:47:11,003 Initializing system.local
       INFO [main] 2013-12-27 20:47:11,003 ColumnFamilyStore.java (line 251) Initializing system.local
       INFO 20:47:11,006 Opening /hadoop/disk1/cassandra/data/system/local/system-local-jb-2478 (5836 bytes)
       INFO [SSTableBatchOpen:1] 2013-12-27 20:47:11,006 SSTableReader.java (line 223) Opening /hadoop/disk1/cassandra/data/system/local/system-local-jb-2478 (5836 bytes)
       INFO 20:47:11,006 Opening /hadoop/disk4/cassandra/data/system/local/system-local-jb-2479 (144 bytes)
       INFO [SSTableBatchOpen:2] 2013-12-27 20:47:11,006 SSTableReader.java (line 223) Opening /hadoop/disk4/cassandra/data/system/local/system-local-jb-2479 (144 bytes)
      ERROR 20:47:12,366 Exception encountered during startup
      java.lang.IllegalArgumentException: bufferSize must be positive
              at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
              at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
              at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:55)
              at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1363)
              at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
              at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1147)
              at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:69)
              at org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1526)
              at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1645)
              at org.apache.cassandra.db.RangeSliceCommand.executeLocally(RangeSliceCommand.java:137)
              at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:236)
              at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:1)
              at org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:255)
              at org.apache.cassandra.db.SystemKeyspace.getUnfinishedCompactions(SystemKeyspace.java:206)
              at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
              at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
              at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
      

      Collecting the logs now, will attach to the issue in a moment.

      Attachments

        1. system.log.gz
          3 kB
          Nikolai Grigoriev
        2. system.log.gz
          11 kB
          Nikolai Grigoriev
        3. cassandra_jstack.txt
          24 kB
          Nikolai Grigoriev
        4. 6531.txt
          2 kB
          Jonathan Ellis

        Activity

          People

            jbellis Jonathan Ellis
            ngrigoriev Nikolai Grigoriev
            Jonathan Ellis
            Pavel Yaskevich
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: