Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-24234

Alerts Are Running For Components Which Are Not Installed

    XMLWordPrintableJSON

Details

    Description

      I recently noticed on a cluster that there were alerts being received and discarded for JournalNode on a cluster which had no JournalNodes installed. I did a quick grep of my own logs, and found a massive amount of alerts that are running for components which are:

      • Not installed on those hosts
      • Not installed anywhere in the cluster
      2018-06-26 19:03:31,054  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert namenode_rpc_latency for an invalid service HDFS and component NAMENODE on host c7402.ambari.apache.org
2018-06-26 19:03:31,054  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org
2018-06-26 19:03:31,997  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:04:25,730  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_webui for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:04:27,011  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:04:30,731  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert nodemanager_health_summary for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:04:31,069  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org
2018-06-26 19:05:30,765  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_webui for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:05:30,767  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert nodemanager_health_summary for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:05:31,090  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert namenode_rpc_latency for an invalid service HDFS and component NAMENODE on host c7402.ambari.apache.org
2018-06-26 19:05:31,091  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org
2018-06-26 19:05:32,036  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:06:25,775  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_webui for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:06:27,047  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:06:30,779  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert nodemanager_health_summary for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:06:31,104  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org
2018-06-26 19:07:30,815  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_webui for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:07:30,816  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_rpc_latency for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:07:30,821  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert nodemanager_health_summary for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:07:30,821  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_cpu for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:07:31,136  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert namenode_rpc_latency for an invalid service HDFS and component NAMENODE on host c7402.ambari.apache.org
2018-06-26 19:07:31,136  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org
2018-06-26 19:07:32,064  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:08:25,835  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_webui for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:08:27,092  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:08:30,835  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert nodemanager_health_summary for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:08:31,150  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org
2018-06-26 19:09:25,862  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_webui for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:09:25,863  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert nodemanager_health_summary for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:09:31,174  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert namenode_rpc_latency for an invalid service HDFS and component NAMENODE on host c7402.ambari.apache.org
2018-06-26 19:09:31,174  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org
2018-06-26 19:09:32,097  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:10:25,883  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_webui for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:10:25,883  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert nodemanager_health_summary for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:10:27,115  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:10:31,199  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org
2018-06-26 19:11:25,897  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_webui for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:11:25,897  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert nodemanager_health_summary for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:11:27,136  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:11:31,229  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert namenode_rpc_latency for an invalid service HDFS and component NAMENODE on host c7402.ambari.apache.org
2018-06-26 19:11:31,229  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org
2018-06-26 19:16:27,291  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:16:30,970  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_webui for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:16:30,971  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_rpc_latency for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:16:30,974  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert nodemanager_health_summary for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:16:30,992  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_cpu for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:16:31,310  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert namenode_rpc_latency for an invalid service HDFS and component NAMENODE on host c7402.ambari.apache.org
2018-06-26 19:16:31,310  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org
2018-06-26 19:17:25,985  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_webui for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:17:30,987  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_rpc_latency for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:17:30,990  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert nodemanager_health_summary for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:17:30,990  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_cpu for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:17:31,324  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert namenode_rpc_latency for an invalid service HDFS and component NAMENODE on host c7402.ambari.apache.org
2018-06-26 19:17:31,324  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org
2018-06-26 19:17:32,274  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:18:26,002  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_webui for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:18:26,004  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert nodemanager_health_summary for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:18:27,294  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:18:31,332  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org
2018-06-26 19:19:26,020  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_webui for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:19:26,021  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert nodemanager_health_summary for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:19:27,327  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:19:31,354  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert namenode_rpc_latency for an invalid service HDFS and component NAMENODE on host c7402.ambari.apache.org
2018-06-26 19:19:31,354  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org
2018-06-26 19:20:26,051  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert yarn_resourcemanager_webui for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:20:26,052  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert nodemanager_health_summary for an invalid service YARN and component RESOURCEMANAGER on host c7403.ambari.apache.org
2018-06-26 19:20:27,354  WARN [alert-event-bus-2] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7401.ambari.apache.org
2018-06-26 19:20:31,370  WARN [alert-event-bus-1] AlertReceivedListener:576 - Unable to process alert journalnode_process for an invalid service HDFS and component JOURNALNODE on host c7402.ambari.apache.org

      STR:

      • Install a simple cluster with ZK, HDFS, and YARN from HDP 2.6
      • Monitor the logs and observe entries for components which are not installed

      Attachments

        Activity

          People

            amagyar Attila Magyar
            amagyar Attila Magyar
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 2h
                2h