[CASSANDRA-11426] End of File Exception (EOF) on startup due to HintsDescriptor read failure

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate

Environment

Debian Jesse patch current, Cassandra 3.0.3

• Severity: Normal
• Assignee: Aleksey Yeschenko
• Reporter: Jason Kania

Authors

Aleksey Yeschenko

• Created: 24/Mar/16 15:04
• Updated: 16/Apr/19 09:30
• Resolved: 19/Mar/17 08:04

Description

I had a node go down and after restart, Cassandra failed to start with the following error popping up. Since the error does not specify the file, it is hard to correct. Running 'nodetool repair' also fails with an unhelpful error, following this error below.

ERROR [main] 2016-03-24 10:49:12,347 CassandraDaemon.java:692 - Exception encountered during startup
org.apache.cassandra.io.FSReadError: java.io.EOFException
at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:142) ~[apache-cassandra-3.0.3.jar:3.0.3]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_74]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_74]
at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[na:1.8.0_74]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[na:1.8.0_74]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_74]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_74]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_74]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_74]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_74]
at org.apache.cassandra.hints.HintsCatalog.load(HintsCatalog.java:65) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.hints.HintsService.<init>(HintsService.java:87) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.hints.HintsService.<clinit>(HintsService.java:62) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.service.StorageProxy.<clinit>(StorageProxy.java:120) ~[apache-cassandra-3.0.3.jar:3.0.3]
at java.lang.Class.forName0(Native Method) ~[na:1.8.0_74]
at java.lang.Class.forName(Class.java:264) ~[na:1.8.0_74]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:625) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:610) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:333) [apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679) [apache-cassandra-3.0.3.jar:3.0.3]
Caused by: java.io.EOFException: null
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:803) ~[na:1.8.0_74]
at org.apache.cassandra.hints.HintsDescriptor.deserialize(HintsDescriptor.java:237) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:138) ~[apache-cassandra-3.0.3.jar:3.0.3]
... 20 common frames omitted

nodetool repair error

root@bull:~# nodetool repair
error: null
– StackTrace –
java.lang.AssertionError
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:105)
at org.apache.cassandra.service.StorageService.getRangesForEndpoint(StorageService.java:3234)
at org.apache.cassandra.service.StorageService.getLocalRanges(StorageService.java:230)
at org.apache.cassandra.service.StorageService.repairAsync(StorageService.java:2935)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Issue Links

• duplicates: CASSANDRA-12728 Handling partially written hint files