Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-25101

HBase metrics not available

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.7.1
    • Fix Version/s: None
    • Component/s: ambari-metrics
    • Labels:
      None

      Description

      TL;DR: I believe this is a bug introduced by the change of the class responsible for fetching the available AMS Collectors. This bug occurs when using more than one AMS Collector, details below.

      -------------------------------------------------------------------------
      This issue affects all metrics that use org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.

      After increasing the verbosity of loggin to DEBUG for the class org.apache.hadoop.metrics2,*, it is possible to see that the metrics are not being sent due a malformed URI in the connection string:

      The malformed URI is composed by both AMS Collectors adresses separated by comma, so I removed one of them as a test and restarted the services. After this no more errors were shown in the logs and the metrics were sent to the AMS DB (they weren't present before). Also, the dashboards started to show some graphs.

      Since this configuration was working in the previous versions (HDP 2.6 + Ambari 2.5) and, according to documentation, is still supported[1] with the new versions (HDP 3.0 + Ambari 2.7) I checked the code of HadoopTimelineMetricsSink[2] to verify what could be causing this. By analyzing the code and correlating with the error messages in the log, we can see that the connection process is handled by another class: AbstractTimelineMetricsSink.

      The AbstractTimelineMetricsSink had modifications from Ambari 2.5[3] to Ambari 2.7[4]. The following snippet highlights the main differences in the connection handling methods used by HadoopTimelineMetricsSink.

      Based on that, I believe this is a bug introduced by the change of AbstractTimelinetMetricsSink class. This bug occurs when using more than one AMS Collector.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              oktaytncy oktay tuncay
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: