Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7849

Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 2.0.11, 2.1.1
    • None
    • None

    Description

      From time to time (actually quite frequently) we get error messages in the server logs like this

      ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118 ErrorMessage.java (line 222) Unexpected exception during request
      java.io.IOException: Connection reset by peer
              at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
              at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
              at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
              at sun.nio.ch.IOUtil.read(IOUtil.java:192)
              at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
              at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      

      These particular cases are almost certainly problems with the client driver, client machine, client process, however after the fact this particular exception is practically impossible to debug because there is no indication in the underlying JVM/netty exception of who the peer was. I should note we have lots of different types of applications running against the cluster so it is very hard to correlate these to anything

      Attachments

        1. cassandra-1.2-7849.txt
          3 kB
          graham sanderson
        2. cassandra-1.2-7849_v2.txt
          4 kB
          graham sanderson
        3. cassandra-1.2-7849_v3.txt
          6 kB
          graham sanderson

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            graham.sanderson graham sanderson Assign to me
            graham sanderson graham sanderson
            graham sanderson
            Tom Hobbs
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment