[AMBARI-16093] LogSearch Integration creates too many error logs during connection failures

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: ambari-server
• Assignee: Bob Nettleton
• Reporter: Bob Nettleton
• Created: 25/Apr/16 15:11
• Updated: 25/Apr/16 22:03
• Resolved: 25/Apr/16 18:41

Description

The Ambari LogSearch integration layer is currently creating too many repeated error log messages in ambari-server.log when a connection attempt fails.

Hundreds of instances of the following log message:

2016-04-20 14:57:13,000 ERROR qtp-ambari-client-324 LoggingSearchPropertyProvider - Error occurred while making request to LogSearch service, unable to populate logging properties on this resource 

appear when a connection attempt to the LogSearch service fails.

This problem can occur when:

1. The LogSearch service crashes or is unexpectedly brought down
2. There is a lag time during which a crashed service is still considered to be in the STARTED state, which is the state checked by the LoggingSearchPropertyProvider to determine if a connection should be attempted.
3. This causes the LogSearch integration code to log many connection attempt failures un-necessarily, since only one instance of this message is necessary.

The LogSearch integration code should use a static counter to determine how many times a given error log should appear.

I'm working on a fix for this, and will submit a patch shortly.