Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7849

Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 2.0.11, 2.1.1
    • Component/s: None
    • Labels:
      None

      Description

      From time to time (actually quite frequently) we get error messages in the server logs like this

      ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118 ErrorMessage.java (line 222) Unexpected exception during request
      java.io.IOException: Connection reset by peer
              at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
              at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
              at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
              at sun.nio.ch.IOUtil.read(IOUtil.java:192)
              at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
              at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      

      These particular cases are almost certainly problems with the client driver, client machine, client process, however after the fact this particular exception is practically impossible to debug because there is no indication in the underlying JVM/netty exception of who the peer was. I should note we have lots of different types of applications running against the cluster so it is very hard to correlate these to anything

        Attachments

        1. cassandra-1.2-7849_v3.txt
          6 kB
          graham sanderson
        2. cassandra-1.2-7849_v2.txt
          4 kB
          graham sanderson
        3. cassandra-1.2-7849.txt
          3 kB
          graham sanderson

          Issue Links

            Activity

              People

              • Assignee:
                graham.sanderson graham sanderson
                Reporter:
                graham sanderson graham sanderson
                Authors:
                graham sanderson
                Reviewers:
                Tom Hobbs
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: