Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14080

Handle incompletely written hint descriptors during startup

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Severity:
      Low

      Description

      Continuation of CASSANDRA-12728 bug.

      Problem: Cassandra didn't start due to 0 size hints files

      Log form v3.0.14:

      INFO  [main] 2017-11-28 19:10:13,554 StorageService.java:575 - Cassandra version: 3.0.14
      INFO  [main] 2017-11-28 19:10:13,555 StorageService.java:576 - Thrift API version: 20.1.0
      INFO  [main] 2017-11-28 19:10:13,555 StorageService.java:577 - CQL supported versions: 3.4.0 (default: 3.4.0)
      ERROR [main] 2017-11-28 19:10:13,592 CassandraDaemon.java:710 - Exception encountered during startup
      org.apache.cassandra.io.FSReadError: java.io.EOFException
              at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:142) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_141]
              at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_141]
              at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[na:1.8.0_141]
              at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[na:1.8.0_141]
              at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_141]
              at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_141]
              at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_141]
              at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_141]
              at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_141]
              at org.apache.cassandra.hints.HintsCatalog.load(HintsCatalog.java:65) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.hints.HintsService.<init>(HintsService.java:88) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.hints.HintsService.<clinit>(HintsService.java:63) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.service.StorageProxy.<clinit>(StorageProxy.java:121) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at java.lang.Class.forName0(Native Method) ~[na:1.8.0_141]
              at java.lang.Class.forName(Class.java:264) ~[na:1.8.0_141]
              at org.apache.cassandra.service.StorageService.initServer(StorageService.java:585) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.service.StorageService.initServer(StorageService.java:570) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346) [apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569) [apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697) [apache-cassandra-3.0.14.jar:3.0.14]
      Caused by: java.io.EOFException: null
              at java.io.RandomAccessFile.readInt(RandomAccessFile.java:803) ~[na:1.8.0_141]
              at org.apache.cassandra.hints.HintsDescriptor.deserialize(HintsDescriptor.java:237) ~[apache-cassandra-3.0.14.jar:3.0.14]
              at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:138) ~[apache-cassandra-3.0.14.jar:3.0.14]
              ... 20 common frames omitted
      

      After several 0 size hints files deletion Cassandra started successfully.

      Jeff Jirsa added a comment - Yesterday
      Aleksandr Ivanov can you open a new JIRA and link it back to this one? It's possible that the original patch didn't consider 0 byte files (I don't have time to go back and look at the commit, and it was long enough ago that I've forgotten) - were all of your files 0 bytes?

      Not all, 8..10 hints files were with 0 size.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              alourie Alex Lourie Assign to me
              Reporter:
              alekiv Aleksandr Ivanov
              Authors:
              Alex Lourie
              Reviewers:
              Aleksey Yeschenko

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment