Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-12924

GraphiteReporter does not reconnect if graphite restarts

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 3.11.3
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal
    • Since Version:

      Description

      Seems like GraphiteReporter does not reconnect after graphite is restarted. The consequence is complete loss of reported metrics until Cassandra restarts. Logs show this every minute:

      WARN  [metrics-graphite-reporter-1-thread-1] 2016-11-17 10:06:26,549 GraphiteReporter.java:179 - Unable to report to Graphite
      java.net.SocketException: Broken pipe
      	at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_91]
      	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0_91]
      	at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_91]
      	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) ~[na:1.8.0_91]
      	at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) ~[na:1.8.0_91]
      	at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) ~[na:1.8.0_91]
      	at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207) ~[na:1.8.0_91]
      	at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129) ~[na:1.8.0_91]
      	at java.io.BufferedWriter.write(BufferedWriter.java:230) ~[na:1.8.0_91]
      	at java.io.Writer.write(Writer.java:157) ~[na:1.8.0_91]
      	at com.codahale.metrics.graphite.Graphite.send(Graphite.java:130) ~[metrics-graphite-3.1.0.jar:3.1.0]
      	at com.codahale.metrics.graphite.GraphiteReporter.reportGauge(GraphiteReporter.java:283) ~[metrics-graphite-3.1.0.jar:3.1.0]
      	at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:158) ~[metrics-graphite-3.1.0.jar:3.1.0]
      	at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) [metrics-core-3.1.0.jar:3.1.0]
      	at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) [metrics-core-3.1.0.jar:3.1.0]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_91]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_91]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_91]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
      WARN  [metrics-graphite-reporter-1-thread-1] 2016-11-17 10:06:26,549 GraphiteReporter.java:183 - Error closing Graphite
      java.net.SocketException: Broken pipe
      	at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_91]
      	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0_91]
      	at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_91]
      	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) ~[na:1.8.0_91]
      	at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) ~[na:1.8.0_91]
      	at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) ~[na:1.8.0_91]
      	at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207) ~[na:1.8.0_91]
      	at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129) ~[na:1.8.0_91]
      	at java.io.BufferedWriter.write(BufferedWriter.java:230) ~[na:1.8.0_91]
      	at java.io.Writer.write(Writer.java:157) ~[na:1.8.0_91]
      	at com.codahale.metrics.graphite.Graphite.send(Graphite.java:130) ~[metrics-graphite-3.1.0.jar:3.1.0]
      	at com.codahale.metrics.graphite.GraphiteReporter.reportGauge(GraphiteReporter.java:283) ~[metrics-graphite-3.1.0.jar:3.1.0]
      	at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:158) ~[metrics-graphite-3.1.0.jar:3.1.0]
      	at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) [metrics-core-3.1.0.jar:3.1.0]
      	at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) [metrics-core-3.1.0.jar:3.1.0]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_91]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_91]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_91]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mck Michael Semb Wever
                Reporter:
                ostefano Stefano Ortolani
                Authors:
                Michael Semb Wever
              • Votes:
                2 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: