[CASSANDRA-7849] Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.0.11, 2.1.1
• Assignee: graham sanderson
• Reporter: graham sanderson

Authors

graham sanderson

Reviewers

Tom Hobbs

• Created: 29/Aug/14 22:58
• Updated: 16/Apr/19 09:31
• Resolved: 18/Sep/14 17:37

Description

From time to time (actually quite frequently) we get error messages in the server logs like this

ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118 ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

These particular cases are almost certainly problems with the client driver, client machine, client process, however after the fact this particular exception is practically impossible to debug because there is no indication in the underlying JVM/netty exception of who the peer was. I should note we have lots of different types of applications running against the cluster so it is very hard to correlate these to anything

Issue Links

• relates to: CASSANDRA-6424 Millions of Message.java (line 342) Got an IOException in internalRead messages on each node of cluster
• relates to: CASSANDRA-7612 java.io.IOException: Connection reset by peer