Details
Trivial Description
First some background:
We were directed to retain audit/access records "forever" (technically 7 years but that is basically forever in electronic log time).
Each Hadoop component generates local audit logs as per their log4j settings. In our production system these logs would frequently fill up the disk. At first we would just compress them in place but that only works for so long and there was no redundancy with local disk storage. In others words, no long term plan.
We started to discuss moving them to HDFS or a different storage solution. One of our team members pointed out the Ranger plugins are already logging the "same data" into HDFS.
Probably after several meeting with the higher-ups, using Ranger logs as the record truth was approved. Components log4j settings were updated to purge data automatically.
Purging local logs felt like operating with out a safety net.
Thought it we be good to check that Ranger was successful logging to HDFS each day. Should mention this is a kerberized cluster, not that anything ever goes wrong with kerberos.
Checking this would have certainly been possible with a shell script, but we have been pushing to centralize warning/alerts in Ambari. And so an Ambari alert python script to check on Ranger Logging Health was crafted.
For the most part the alert was modeled after some of the hive alerts.
At the moment it just checks that the daily /ranger/audit/<component> HDFS directory has been created.
I am attaching the host script and the alert.json for HDFS and Knox components.
In the alert.json, service_name and component_name should be set to local values.
Everything else should "work out of the box".