Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7849

Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 2.0.11, 2.1.1
    • None
    • None

    Description

      From time to time (actually quite frequently) we get error messages in the server logs like this

      ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118 ErrorMessage.java (line 222) Unexpected exception during request
      java.io.IOException: Connection reset by peer
              at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
              at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
              at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
              at sun.nio.ch.IOUtil.read(IOUtil.java:192)
              at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
              at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      

      These particular cases are almost certainly problems with the client driver, client machine, client process, however after the fact this particular exception is practically impossible to debug because there is no indication in the underlying JVM/netty exception of who the peer was. I should note we have lots of different types of applications running against the cluster so it is very hard to correlate these to anything

      Attachments

        1. cassandra-1.2-7849_v3.txt
          6 kB
          graham sanderson
        2. cassandra-1.2-7849_v2.txt
          4 kB
          graham sanderson
        3. cassandra-1.2-7849.txt
          3 kB
          graham sanderson

        Issue Links

          Activity

            People

              graham.sanderson graham sanderson
              graham sanderson graham sanderson
              graham sanderson
              Tom Hobbs
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: