Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
2.5.0
-
None
Description
The LogSearch Integration layer in Ambari typically caches logging metadata for each host/component combination in a given cluster. This cache is configurable, and is typically a long-lived cache, since the logging metadata is generally not likely to change.
This issue was found during a test of LogSearch at a large scale, with a cluster of over 800 nodes.
it was discovered that the LogSearch Integration layer continues to make repeated requests for logging metadata for components that are not properly configured in LogSearch.
The LogSearch portal logs show the following repeated requests on the "Perf" cluster:
{"level":"INFO","file":"SolrDaoBase.java","thread_name":"qtp1018937824-280","line_number":61,"logger_name":"org.apache.ambari.logsearch.dao.SolrDaoBase","logtime":"1487951210778","log_message":"Solr query will be processed: q=*:*&start=0&rows=1&fq=host:perf\\-d\\-86.c.pramod\\-thangali.internal&fq=type:hst_agent&fq=logtime:[*+TO+*]&sort=logtime+desc,seq_num+desc"}
{"level":"INFO","file":"SolrDaoBase.java","thread_name":"qtp1018937824-273","line_number":61,"logger_name":"org.apache.ambari.logsearch.dao.SolrDaoBase","logtime":"1487951210797","log_message":"Solr query will be processed: q=*:*&start=0&rows=1&fq=host:perf\\-k\\-24.c.pramod\\-thangali.internal&fq=type:hst_agent&fq=logtime:[*+TO+*]&sort=logtime+desc,seq_num+desc"}
{"level":"INFO","file":"SolrDaoBase.java","thread_name":"qtp1018937824-275","line_number":61,"logger_name":"org.apache.ambari.logsearch.dao.SolrDaoBase","logtime":"1487951210810","log_message":"Solr query will be processed: q=*:*&start=0&rows=1&fq=host:perf\\-d\\-88.c.pramod\\-thangali.internal&fq=type:hst_agent&fq=logtime:[*+TO+*]&sort=logtime+desc,seq_num+desc"}
The SmartSense component, managed outside of Ambari, includes logging component IDs the SmartSense stack definition, but does not include the proper metadata to allow LogSearch Portal to properly handle the SmartSense logs.
In the current implementation of the integration layer, since the SmartSense components are never found, the integration will repeatedly check the cache, fail that check, and attempt a remote call to obtain the metadata. This in turn can cause LogSearch to execute repeated, and un-necessary, queries against the Solr database, which could potentially be a problem in larger deployments.
The LogSearch Integration in Ambari should be modified to have a maximum number of failed attempts to tolerate. Having passed a maximum number of failed attempts to obtain the logging metadata for a given component, the integration code should refrain from further attempts to contact the LogSearch portal for this component, and log a warning to indicate that the requested component is not properly handled by LogSearch.
This will probably need to be fixed in:
org.apache.ambari.server.controller.logging.LogSearchDataRetrievalService
