Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14080

Handle incompletely written hint descriptors during startup

    XMLWordPrintableJSON

    Details

    • Severity:
      Low

      Description

      Continuation of CASSANDRA-12728 bug.

      Problem: Cassandra didn't start due to 0 size hints files

      Log form v3.0.14:

      INFO  [main] 2017-11-28 19:10:13,554 StorageService.java:575 - Cassandra version: 3.0.14
      INFO  [main] 2017-11-28 19:10:13,555 StorageService.java:576 - Thrift API version: 20.1.0
      INFO  [main] 2017-11-28 19:10:13,555 StorageService.java:577 - CQL supported versions: 3.4.0 (default: 3.4.0)
      ERROR [main] 2017-11-28 19:10:13,592 CassandraDaemon.java:710 - Exception encountered during startup
      org.apache.cassandra.io.FSReadError: java.io.EOFException
              at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:142) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_141]
              at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_141]
              at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[na:1.8.0_141]
              at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[na:1.8.0_141]
              at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_141]
              at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_141]
              at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_141]
              at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_141]
              at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_141]
              at org.apache.cassandra.hints.HintsCatalog.load(HintsCatalog.java:65) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.hints.HintsService.<init>(HintsService.java:88) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.hints.HintsService.<clinit>(HintsService.java:63) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.service.StorageProxy.<clinit>(StorageProxy.java:121) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at java.lang.Class.forName0(Native Method) ~[na:1.8.0_141]
              at java.lang.Class.forName(Class.java:264) ~[na:1.8.0_141]
              at org.apache.cassandra.service.StorageService.initServer(StorageService.java:585) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.service.StorageService.initServer(StorageService.java:570) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346) [apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569) [apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697) [apache-cassandra-3.0.14.jar:3.0.14]
      Caused by: java.io.EOFException: null
              at java.io.RandomAccessFile.readInt(RandomAccessFile.java:803) ~[na:1.8.0_141]
              at org.apache.cassandra.hints.HintsDescriptor.deserialize(HintsDescriptor.java:237) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:138) ~[apache-cassandra-3.0.14.jar:3.0.14]
              ... 20 common frames omitted
      

      After several 0 size hints files deletion Cassandra started successfully.

      Jeff Jirsa added a comment - Yesterday
      Aleksandr Ivanov can you open a new JIRA and link it back to this one? It's possible that the original patch didn't consider 0 byte files (I don't have time to go back and look at the commit, and it was long enough ago that I've forgotten) - were all of your files 0 bytes?

      Not all, 8..10 hints files were with 0 size.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                alourie Alex Lourie
                Reporter:
                alekiv Aleksandr Ivanov
                Authors:
                Alex Lourie
                Reviewers:
                Aleksey Yeschenko
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: