Details

    • Type: New Feature New Feature
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 3.5.3
    • Component/s: None
    • Labels:
      None

      Description

      Lots of users have had questions on debugging which client changed what znode and what updates went through a znode. We should add audit logging as in Hadoop (look at Namenode Audit logging) to log which client changed what in the zookeeper servers. This could just be a log4j audit logger.

      1. ZOOKEEPER-1260-01.patch
        104 kB
        Arshad Mohammad
      2. zookeeperAuditLogs.pdf
        40 kB
        Arshad Mohammad

        Issue Links

          Activity

          Hide
          zhihai xu added a comment -

          This sounds like a very useful feature for production system.

          Show
          zhihai xu added a comment - This sounds like a very useful feature for production system.
          Hide
          Edward Ribeiro added a comment -

          Please, refer to my comment at ZOOKEEPER-2287 (duplicate of this one) just to check if it's relevant or not.

          Cheers!

          Show
          Edward Ribeiro added a comment - Please, refer to my comment at ZOOKEEPER-2287 (duplicate of this one) just to check if it's relevant or not. Cheers!
          Hide
          Arshad Mohammad added a comment -

          Edward Ribeiro , regarding your comments on ZOOKEEPER-2287.
          First of all, Thanks for sharing your thoughts. But I have completely different opinion on this..

          1. JMX matrices and audit logs serve completely different purposes. Zookeeper JMX metrics give the current state of the system.
            Where as the audit log, in general, give the history of the operations that change the system state. So adding more matrices can not replace audit log
          2. Audit logs does not log all the operations, they log only the operations which change the state of the system. So the amount of audit log compared to general log is very very less.
          3. When the audit log is disabled, the performance impact is negligible. But when audit log is enabled offcourse there will be slight performace degradation, but that is optional to user whether they want the audit log or slightly better performance
          Show
          Arshad Mohammad added a comment - Edward Ribeiro , regarding your comments on ZOOKEEPER-2287 . First of all, Thanks for sharing your thoughts. But I have completely different opinion on this.. JMX matrices and audit logs serve completely different purposes. Zookeeper JMX metrics give the current state of the system. Where as the audit log, in general, give the history of the operations that change the system state. So adding more matrices can not replace audit log Audit logs does not log all the operations, they log only the operations which change the state of the system. So the amount of audit log compared to general log is very very less. When the audit log is disabled, the performance impact is negligible. But when audit log is enabled offcourse there will be slight performace degradation, but that is optional to user whether they want the audit log or slightly better performance
          Hide
          Edward Ribeiro added a comment - - edited

          JMX matrices and audit logs serve completely different purposes.

          Yup, I know, but I was thinking about exporting those metrics to systems like Graphite or Nagios that can keep a record of ops exposed (given the right adapters), aggregate the data and show them in fancy dashboards. But you right: they are complementary. I would even go further to suggest that we could have JMX commands to enable/disable the audit log.

          Audit logs does not log all the operations, they log only the operations which change the state of the system.

          Even so, this can be a LOT of operations (state changes). THIS, in fact is my only point regarding this issue, but I am all favour about the prospect of adding an audit log. Even on previous comment about it (sorry, if I was unclear about that).

          When the audit log is disabled, the performance impact is negligible. But when audit log is enabled offcourse there will be slight performace degradation

          Sure. Let's do this and measure his performance degradation under a high load of write ops, so that users can be aware of its impact.

          TL;DR: I am +1 about adding an audit log, we certainly need this.

          Show
          Edward Ribeiro added a comment - - edited JMX matrices and audit logs serve completely different purposes. Yup, I know, but I was thinking about exporting those metrics to systems like Graphite or Nagios that can keep a record of ops exposed (given the right adapters), aggregate the data and show them in fancy dashboards. But you right: they are complementary. I would even go further to suggest that we could have JMX commands to enable/disable the audit log. Audit logs does not log all the operations, they log only the operations which change the state of the system. Even so, this can be a LOT of operations (state changes) . THIS, in fact is my only point regarding this issue , but I am all favour about the prospect of adding an audit log. Even on previous comment about it (sorry, if I was unclear about that). When the audit log is disabled, the performance impact is negligible. But when audit log is enabled offcourse there will be slight performace degradation Sure. Let's do this and measure his performance degradation under a high load of write ops, so that users can be aware of its impact. TL;DR: I am +1 about adding an audit log, we certainly need this.
          Hide
          Arshad Mohammad added a comment -

          Submitting the implementation of ZooKeeper audit log feature. Please refer attached zookeeperAuditLogs.pdf for implementation details.

          Show
          Arshad Mohammad added a comment - Submitting the implementation of ZooKeeper audit log feature. Please refer attached zookeeperAuditLogs.pdf for implementation details.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12823635/ZOOKEEPER-1260-01.patch
          against trunk revision 1756262.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3369//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12823635/ZOOKEEPER-1260-01.patch against trunk revision 1756262. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3369//console This message is automatically generated.

            People

            • Assignee:
              Arshad Mohammad
              Reporter:
              Mahadev konar
            • Votes:
              6 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:

                Development