Details

    • Type: New Feature
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 3.5.4, 3.6.0
    • Component/s: None
    • Labels:
      None

      Description

      Lots of users have had questions on debugging which client changed what znode and what updates went through a znode. We should add audit logging as in Hadoop (look at Namenode Audit logging) to log which client changed what in the zookeeper servers. This could just be a log4j audit logger.

      1. ZOOKEEPER-1260-01.patch
        104 kB
        Mohammad Arshad
      2. zookeeperAuditLogs.pdf
        40 kB
        Mohammad Arshad

        Issue Links

          Activity

          Hide
          zxu zhihai xu added a comment -

          This sounds like a very useful feature for production system.

          Show
          zxu zhihai xu added a comment - This sounds like a very useful feature for production system.
          Hide
          eribeiro Edward Ribeiro added a comment -

          Please, refer to my comment at ZOOKEEPER-2287 (duplicate of this one) just to check if it's relevant or not.

          Cheers!

          Show
          eribeiro Edward Ribeiro added a comment - Please, refer to my comment at ZOOKEEPER-2287 (duplicate of this one) just to check if it's relevant or not. Cheers!
          Hide
          arshad.mohammad Mohammad Arshad added a comment -

          Edward Ribeiro , regarding your comments on ZOOKEEPER-2287.
          First of all, Thanks for sharing your thoughts. But I have completely different opinion on this..

          1. JMX matrices and audit logs serve completely different purposes. Zookeeper JMX metrics give the current state of the system.
            Where as the audit log, in general, give the history of the operations that change the system state. So adding more matrices can not replace audit log
          2. Audit logs does not log all the operations, they log only the operations which change the state of the system. So the amount of audit log compared to general log is very very less.
          3. When the audit log is disabled, the performance impact is negligible. But when audit log is enabled offcourse there will be slight performace degradation, but that is optional to user whether they want the audit log or slightly better performance
          Show
          arshad.mohammad Mohammad Arshad added a comment - Edward Ribeiro , regarding your comments on ZOOKEEPER-2287 . First of all, Thanks for sharing your thoughts. But I have completely different opinion on this.. JMX matrices and audit logs serve completely different purposes. Zookeeper JMX metrics give the current state of the system. Where as the audit log, in general, give the history of the operations that change the system state. So adding more matrices can not replace audit log Audit logs does not log all the operations, they log only the operations which change the state of the system. So the amount of audit log compared to general log is very very less. When the audit log is disabled, the performance impact is negligible. But when audit log is enabled offcourse there will be slight performace degradation, but that is optional to user whether they want the audit log or slightly better performance
          Hide
          eribeiro Edward Ribeiro added a comment - - edited

          JMX matrices and audit logs serve completely different purposes.

          Yup, I know, but I was thinking about exporting those metrics to systems like Graphite or Nagios that can keep a record of ops exposed (given the right adapters), aggregate the data and show them in fancy dashboards. But you right: they are complementary. I would even go further to suggest that we could have JMX commands to enable/disable the audit log.

          Audit logs does not log all the operations, they log only the operations which change the state of the system.

          Even so, this can be a LOT of operations (state changes). THIS, in fact is my only point regarding this issue, but I am all favour about the prospect of adding an audit log. Even on previous comment about it (sorry, if I was unclear about that).

          When the audit log is disabled, the performance impact is negligible. But when audit log is enabled offcourse there will be slight performace degradation

          Sure. Let's do this and measure his performance degradation under a high load of write ops, so that users can be aware of its impact.

          TL;DR: I am +1 about adding an audit log, we certainly need this.

          Show
          eribeiro Edward Ribeiro added a comment - - edited JMX matrices and audit logs serve completely different purposes. Yup, I know, but I was thinking about exporting those metrics to systems like Graphite or Nagios that can keep a record of ops exposed (given the right adapters), aggregate the data and show them in fancy dashboards. But you right: they are complementary. I would even go further to suggest that we could have JMX commands to enable/disable the audit log. Audit logs does not log all the operations, they log only the operations which change the state of the system. Even so, this can be a LOT of operations (state changes) . THIS, in fact is my only point regarding this issue , but I am all favour about the prospect of adding an audit log. Even on previous comment about it (sorry, if I was unclear about that). When the audit log is disabled, the performance impact is negligible. But when audit log is enabled offcourse there will be slight performace degradation Sure. Let's do this and measure his performance degradation under a high load of write ops, so that users can be aware of its impact. TL;DR: I am +1 about adding an audit log, we certainly need this.
          Hide
          arshad.mohammad Mohammad Arshad added a comment -

          Submitting the implementation of ZooKeeper audit log feature. Please refer attached zookeeperAuditLogs.pdf for implementation details.

          Show
          arshad.mohammad Mohammad Arshad added a comment - Submitting the implementation of ZooKeeper audit log feature. Please refer attached zookeeperAuditLogs.pdf for implementation details.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12823635/ZOOKEEPER-1260-01.patch
          against trunk revision 1756262.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3369//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12823635/ZOOKEEPER-1260-01.patch against trunk revision 1756262. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3369//console This message is automatically generated.
          Hide
          mathias.kluba mathias kluba added a comment -

          It will be good to create a "plugin" model to be able to send audits to Ranger: https://issues.apache.org/jira/browse/RANGER-924

          Show
          mathias.kluba mathias kluba added a comment - It will be good to create a "plugin" model to be able to send audits to Ranger: https://issues.apache.org/jira/browse/RANGER-924
          Hide
          fpj Flavio Junqueira added a comment -

          Should we revive this patch? It seems to be stale.

          Show
          fpj Flavio Junqueira added a comment - Should we revive this patch? It seems to be stale.
          Hide
          arshad.mohammad Mohammad Arshad added a comment -

          Thanks Flavio Junqueira for showing interest in this feature. I will re-base it soon and raise a PR.

          Show
          arshad.mohammad Mohammad Arshad added a comment - Thanks Flavio Junqueira for showing interest in this feature. I will re-base it soon and raise a PR.

            People

            • Assignee:
              arshad.mohammad Mohammad Arshad
              Reporter:
              mahadev Mahadev konar
            • Votes:
              6 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:

                Development