[AMBARI-25400] Issue while determining live collector in case of HA

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.2, 2.7.4
• Fix Version/s: 2.8.0, 2.7.5
• Component/s: ambari-metrics
• Labels: pull-request-available
• Assignee: Éberhardt Péter
• Reporter: amarnath reddy pappu
• Created: 23/Oct/19 21:48
• Updated: 07/Nov/22 19:19
• Resolved: 30/Oct/19 13:33

Description

If collector throws http://collectorhost:port//ws/v1/timeline/metrics/livenodes 500 error then sink is unable to determine live/healthy collector.

 

sink will try to connect to another collector only if it is not reachable/IOException to first collector.  if any other response code other than 200 then still it should consider as 1st Collector not reachable.

 

https://github.com/apache/ambari/blob/release-2.7.4/ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/AbstractTimelineMetricsSink.java#L629

 

 Possible fix

 

      if (responseCode == 200) {
        try (InputStream in = connection.getInputStream()) {
          StringWriter writer = new StringWriter();
          IOUtils.copy(in, writer);
          try {
            collectors = gson.fromJson(writer.toString(), new TypeToken<List<String>>(){}.getType());
          } catch (JsonSyntaxException jse) {
            // Swallow this at the behest of still trying to POST
            LOG.debug("Exception deserializing the json data on live " +
              "collector nodes.", jse);
          }
        }
      } else if (responseCode == 500){
        String warnMsg = "Unable to connect to collector to find live nodes, Internal server error";
        throw new MetricCollectorUnavailableException(warnMsg);
      }

 

Issue Links

• links to: GitHub Pull Request #51
• links to: GitHub Pull Request #3108