Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15008

Metrics sinks may emit too frequently if multiple sink periods are configured

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.0, 3.0.0-beta1
    • Fix Version/s: 3.1.0
    • Component/s: metrics
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Previously if multiple metrics sinks were configured with different periods, they may emit more frequently than configured, at a period as low as the GCD of the configured periods. This change makes all metrics sinks emit at their configured period.

      Description

      If there are multiple metrics sink periods configured, depending on what those periods are, some sinks may emit too frequently. For example with the following:

      hadoop-metrics2.properties
      namenode.sink.file10.class=org.apache.hadoop.metrics2.sink.FileSink
      namenode.sink.file5.class=org.apache.hadoop.metrics2.sink.FileSink
      namenode.sink.file10.filename=namenode-metrics_per10.out
      namenode.sink.file5.filename=namenode-metrics_per5.out
      namenode.sink.file10.period=10
      namenode.sink.file5.period=5
      

      I get the following:

      ± for f in namenode-metrics_per*.out; do echo "$f" && grep "metricssystem.MetricsSystem" $f | awk '{last=curr; curr=$1} END { print curr-last }'; done
      namenode-metrics_per10.out
      5000
      namenode-metrics_per5.out
      5000
      

      i.e., for both metrics files, each record is 5000 ms apart, even though one of the sinks has been configured to emit at 10s intervals

        Attachments

          Activity

            People

            • Assignee:
              xkrogen Erik Krogen
              Reporter:
              xkrogen Erik Krogen
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: