Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15008

Metrics sinks may emit too frequently if multiple sink periods are configured

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.2.0, 3.0.0-beta1
    • 3.1.0
    • metrics
    • None
    • Incompatible change, Reviewed
    • Previously if multiple metrics sinks were configured with different periods, they may emit more frequently than configured, at a period as low as the GCD of the configured periods. This change makes all metrics sinks emit at their configured period.

    Description

      If there are multiple metrics sink periods configured, depending on what those periods are, some sinks may emit too frequently. For example with the following:

      hadoop-metrics2.properties
      namenode.sink.file10.class=org.apache.hadoop.metrics2.sink.FileSink
      namenode.sink.file5.class=org.apache.hadoop.metrics2.sink.FileSink
      namenode.sink.file10.filename=namenode-metrics_per10.out
      namenode.sink.file5.filename=namenode-metrics_per5.out
      namenode.sink.file10.period=10
      namenode.sink.file5.period=5
      

      I get the following:

      ± for f in namenode-metrics_per*.out; do echo "$f" && grep "metricssystem.MetricsSystem" $f | awk '{last=curr; curr=$1} END { print curr-last }'; done
      namenode-metrics_per10.out
      5000
      namenode-metrics_per5.out
      5000
      

      i.e., for both metrics files, each record is 5000 ms apart, even though one of the sinks has been configured to emit at 10s intervals

      Attachments

        1. HADOOP-15008.000.patch
          7 kB
          Erik Krogen

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            xkrogen Erik Krogen
            xkrogen Erik Krogen
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment