[CASSANDRA-14080] Handle incompletely written hint descriptors during startup

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 3.0.17, 3.11.3, 4.0-alpha1, 4.0
• Component/s: Consistency/Hints
• Severity: Low
• Assignee: Alex Lourie
• Reporter: Aleksandr Ivanov

Authors

Alex Lourie

Reviewers

Aleksey Yeschenko

• Created: 30/Nov/17 12:06
• Updated: 07/Mar/23 11:52
• Resolved: 29/Mar/18 13:05

Description

Continuation of CASSANDRA-12728 bug.

Problem: Cassandra didn't start due to 0 size hints files

Log form v3.0.14:

INFO  [main] 2017-11-28 19:10:13,554 StorageService.java:575 - Cassandra version: 3.0.14
INFO  [main] 2017-11-28 19:10:13,555 StorageService.java:576 - Thrift API version: 20.1.0
INFO  [main] 2017-11-28 19:10:13,555 StorageService.java:577 - CQL supported versions: 3.4.0 (default: 3.4.0)
ERROR [main] 2017-11-28 19:10:13,592 CassandraDaemon.java:710 - Exception encountered during startup
org.apache.cassandra.io.FSReadError: java.io.EOFException
        at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:142) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_141]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_141]
        at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[na:1.8.0_141]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[na:1.8.0_141]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_141]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_141]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_141]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_141]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_141]
        at org.apache.cassandra.hints.HintsCatalog.load(HintsCatalog.java:65) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.hints.HintsService.<init>(HintsService.java:88) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.hints.HintsService.<clinit>(HintsService.java:63) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.service.StorageProxy.<clinit>(StorageProxy.java:121) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at java.lang.Class.forName0(Native Method) ~[na:1.8.0_141]
        at java.lang.Class.forName(Class.java:264) ~[na:1.8.0_141]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:585) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:570) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346) [apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569) [apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697) [apache-cassandra-3.0.14.jar:3.0.14]
Caused by: java.io.EOFException: null
        at java.io.RandomAccessFile.readInt(RandomAccessFile.java:803) ~[na:1.8.0_141]
        at org.apache.cassandra.hints.HintsDescriptor.deserialize(HintsDescriptor.java:237) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:138) ~[apache-cassandra-3.0.14.jar:3.0.14]
        ... 20 common frames omitted

After several 0 size hints files deletion Cassandra started successfully.

Jeff Jirsa added a comment - Yesterday
Aleksandr Ivanov can you open a new JIRA and link it back to this one? It's possible that the original patch didn't consider 0 byte files (I don't have time to go back and look at the commit, and it was long enough ago that I've forgotten) - were all of your files 0 bytes?

Not all, 8..10 hints files were with 0 size.

Issue Links

• is related to: CASSANDRA-11090 Avoid creating empty hint files
• relates to: CASSANDRA-12728 Handling partially written hint files