[CASSANDRA-3585] Intermittent exceptions seen in cassandra 1.0.5 during Reads.

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 0.8.9, 1.0.6

Environment

RHEL 2.6.32-71.el6.x86_64.
RAM - 28GB
11 CPUs of 2.6GHz

• Severity: Normal
• Assignee: Unassigned
• Reporter: Shantanu
• Created: 07/Dec/11 18:18
• Updated: 16/Apr/19 09:32
• Resolved: 10/Dec/11 15:54

Description

In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs -

ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main]
java.io.IOError: java.io.EOFException
at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126)
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100)
at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81)
at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
... 6 more

RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum.
I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't.

Thanks,
Shantanu