Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13730

After 5 connection failures, yarn stops sending metrics graphite until restarted

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.7.2
    • Fix Version/s: None
    • Component/s: common
    • Labels:
      None
    • Target Version/s:

      Description

      We've had issues in production where metrics stopped. We found the following in the log files:

      2016-09-02 21:44:32,493 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: Error sending metrics to Graphite
      java.net.SocketException: Broken pipe
      at java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
      at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
      at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
      at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
      at java.io.Writer.write(Writer.java:154)
      at org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
      at org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
      at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)

      2016-09-03 00:03:04,335 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: Error sending metrics to Graphite
      java.net.SocketException: Broken pipe
      at java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
      at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
      at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
      at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
      at java.io.Writer.write(Writer.java:154)
      at org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
      at org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
      at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)

      2016-09-03 00:20:35,436 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: Error sending metrics to Graphite
      java.net.SocketException: Connection timed out
      at java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
      at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
      at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
      at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
      at java.io.Writer.write(Writer.java:154)
      at org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
      at org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
      at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
      2016-09-03 00:22:48,862 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: Error sending metrics to Graphite
      java.net.SocketException: Broken pipe
      at java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
      at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
      at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
      at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
      at java.io.Writer.write(Writer.java:154)
      at org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
      at org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
      at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)

      2016-09-03 00:24:00,270 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: Error sending metrics to Graphite
      java.net.SocketException: Broken pipe
      at java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
      at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
      at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
      at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
      at java.io.Writer.write(Writer.java:154)
      at org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
      at org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
      at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
      2016-09-03 00:24:41,987 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 482 Total time for transactions(ms): 9 Number of transactions batched in Syncs: 3 Number of syncs: 355 SyncTimes(ms): 342
      2016-09-03 00:25:39,772 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: Error sending metrics to Graphite
      java.net.SocketException: Connection reset
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:124)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
      at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
      at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
      at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
      at java.io.Writer.write(Writer.java:154)
      at org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
      at org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
      at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)

      The last error was:
      2016-09-03 01:13:51,619 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: Error sending metrics to Graphite
      java.net.SocketException: Broken pipe
      at java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
      at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
      at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
      at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
      at java.io.Writer.write(Writer.java:154)
      at org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
      at org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
      at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
      at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
      2016-09-03 01:15:58,828 ERROR org.apache.hadoop.metrics2.sink.GraphiteSink: Too many connection failures, would not try to connect again.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Sean Young Sean Young
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated: