[AMBARI-20378] LogFeeder: Add de-duplication support

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.5.0
• Fix Version/s: trunk, 2.5.1
• Component/s: ambari-logsearch
• Assignee: Oliver Szabo
• Reporter: Oliver Szabo
• Created: 09/Mar/17 13:22
• Updated: 12/Apr/17 13:16
• Resolved: 22/Mar/17 13:22

Description

Add de-duplication support to logfeeder.
For handling duplications, it will be an LRU cache for every input.
It can be set up at logfeeder.properties (global setting). For example (the values below are the defaults):

logfeeder.cache.enabled=false
logfeeder.cache.size=100
logfeeder.cache.key.field=log_message
logfeeder.cache.dedup.interval=1000
logfeeder.cache.last.dedup.enabled=false

can be overriden in input config block (per monitored file)

{
  "input": [
    {
      ...
      "cache_enabled" : "true",
      "cache_size" : "100",
      "cache_dedup_interval" : "1000",
      "cache_last_dedup_enabled" : "false",
      "cache_key_field" : "log_message"
    }
  ] ...

cache_dedup_interval : if the interval (timestamp - date difference) is not reached between 2 log messages (same content), then the new log will be dropped.
cache_last_dedup_enabled: if its enabled and the new log message is the same as the last one, the new log will be dropped. (dedup interval wont have impact on that feature)
cache_key_field : field of the log message which will be compared and stored in the cache (as keys)

Issue Links

• links to: Review Board