Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.7.1
-
None
-
None
Description
TL;DR: I believe this is a bug introduced by the change of the class responsible for fetching the available AMS Collectors. This bug occurs when using more than one AMS Collector, details below.
-------------------------------------------------------------------------
This issue affects all metrics that use org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.
After increasing the verbosity of loggin to DEBUG for the class org.apache.hadoop.metrics2,*, it is possible to see that the metrics are not being sent due a malformed URI in the connection string:
The malformed URI is composed by both AMS Collectors adresses separated by comma, so I removed one of them as a test and restarted the services. After this no more errors were shown in the logs and the metrics were sent to the AMS DB (they weren't present before). Also, the dashboards started to show some graphs.
Since this configuration was working in the previous versions (HDP 2.6 + Ambari 2.5) and, according to documentation, is still supported[1] with the new versions (HDP 3.0 + Ambari 2.7) I checked the code of HadoopTimelineMetricsSink[2] to verify what could be causing this. By analyzing the code and correlating with the error messages in the log, we can see that the connection process is handled by another class: AbstractTimelineMetricsSink.
The AbstractTimelineMetricsSink had modifications from Ambari 2.5[3] to Ambari 2.7[4]. The following snippet highlights the main differences in the connection handling methods used by HadoopTimelineMetricsSink.
Based on that, I believe this is a bug introduced by the change of AbstractTimelinetMetricsSink class. This bug occurs when using more than one AMS Collector.
