[METRON-170] Ability for metron users to author rules (Queries) to generate alerts without deploying code (Batch Rules Engine)

• Type: Wish
• Status: To Do
• Priority: Critical
• Resolution: Unresolved
• Labels: ForwardLookingEpic elasticsearch hive platform ruleengine rules solr spark
• Assignee: Unassigned
• Reporter: Zafer Bilaloglu
• Created: 17/May/16 20:56
• Updated: 02/Nov/16 16:55

Description

The primary purpose for a rules engine for Apache Metron would be to allow Metron users to author rules that then generate alerts for SIC analysts to investigate. Typical enterprises have hundreds of rules (dozens for each data source) and need the flexibility to alter rules as needed without deploying code to production. Rules would run on a schedule in batch mode and perform predefined action such as generating an alert.

Here are some example rules we'd like to be able to run in metron with the rule syntax written in SQL:

Rule Description	Rule Syntax	Schedule
Mcafee epo log entry notifies us a malware delete failed	Select * from mcafee where event_description = “Malware Delete Failed”	Run every 5 minutes and for new data in the previous 5 minutes.
Multiple malware events for a single user within a short period of time	Select count( * ) as avcount, user from mcafee group by user, dest_ip, os where category like 'av.%' and avcount > 8	Run every 60 minutes for the previous 60 minutes

Users should have a front end to author rules, decide on a schedule, and configure an alert priority, rule description, and the action to perform(alert, e:

Here is a sample mockup:

The batch rules engine would fire recurring queries against data at rest in one of the existing Metron datastores (Solr, Hive, Elasticsearch) that will then perform predefined action such as generating an alert, running a script (python), or kicking off packet capture.