Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7849

Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 2.0.11, 2.1.1
    • Component/s: None
    • Labels:
      None

      Description

      From time to time (actually quite frequently) we get error messages in the server logs like this

      ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118 ErrorMessage.java (line 222) Unexpected exception during request
      java.io.IOException: Connection reset by peer
              at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
              at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
              at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
              at sun.nio.ch.IOUtil.read(IOUtil.java:192)
              at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
              at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      

      These particular cases are almost certainly problems with the client driver, client machine, client process, however after the fact this particular exception is practically impossible to debug because there is no indication in the underlying JVM/netty exception of who the peer was. I should note we have lots of different types of applications running against the cluster so it is very hard to correlate these to anything

        Attachments

        1. cassandra-1.2-7849.txt
          3 kB
          graham sanderson
        2. cassandra-1.2-7849_v2.txt
          4 kB
          graham sanderson
        3. cassandra-1.2-7849_v3.txt
          6 kB
          graham sanderson

        Issue Links

          Activity

            People

            • Assignee:
              graham.sanderson graham sanderson Assign to me
              Reporter:
              graham sanderson graham sanderson
              Authors:
              graham sanderson
              Reviewers:
              Tom Hobbs

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment