ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-1183

Enhance LogFormatter to output additional detail from transaction log

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.4.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Current LogFormatter prints the following information
      ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
      8/15/11 1:55:36 PM PDT session 0x131cf1a236f0014 cxid 0x0 zxid 0xf01 createSession
      8/15/11 1:55:57 PM PDT session 0x131cf1a236f0000 cxid 0x55f zxid 0xf02 setData
      8/15/11 1:56:00 PM PDT session 0x131cf1a236f0015 cxid 0x0 zxid 0xf03 createSession
      ...
      ..
      8/15/11 2:00:33 PM PDT session 0x131cf1a236f001c cxid 0x36 zxid 0xf6b setData
      8/15/11 2:00:33 PM PDT session 0x131cf1a236f0021 cxid 0xa1 zxid 0xf6c create
      8/15/11 2:00:33 PM PDT session 0x131cf1a236f001b cxid 0x3e zxid 0xf6d setData
      8/15/11 2:00:33 PM PDT session 0x131cf1a236f001e cxid 0x3e zxid 0xf6e setData
      8/15/11 2:00:33 PM PDT session 0x131cf1a236f001d cxid 0x41 zxid 0xf6f setData

      Though this is good information, it does not provide additional information like
      createSession: which ip created the session and its time out
      set|get|delete: the path and data
      create: path created and createmode along with data

      We can add additional parameter -detail and provide detailed output of the transaction.

      Outputting data is slightly tricky since we cant print data without understanding the format. We need not print this for now.

      1. ZOOKEEPER-1183.patch
        5 kB
        kishore gopalakrishna

        Activity

        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12495006/ZOOKEEPER-1183.patch
        against trunk revision 1531061.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1676//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12495006/ZOOKEEPER-1183.patch against trunk revision 1531061. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1676//console This message is automatically generated.
        Hide
        kishore gopalakrishna added a comment -

        The patch I have provided adds the ability to print the data if the transaction modifies the data along with the path that change. You can check out the patch and free to either use it, modify it or start from scratch.( It does not handle multi transaction)

        The requirements I added were from the experience I had when we used this patched LogFormatter to provide additional data needed to debug https://issues.apache.org/jira/browse/ZOOKEEPER-1208.

        I dont think its ambitious, its based on whether you want to say its not a bug(I guess thats what you said in that 1208) because the user cant provide additional/relevant data or have the ability to ask user to run a command(which would have been a better thing) and give its output and you can reason about it.

        We had to spend quite some time to actually provide the relevant data needed for the ticket to actually find the issue and be solved. Don't know if its the right thing to expect every one to understand the transaction log to provide the relevant data.

        Show
        kishore gopalakrishna added a comment - The patch I have provided adds the ability to print the data if the transaction modifies the data along with the path that change. You can check out the patch and free to either use it, modify it or start from scratch.( It does not handle multi transaction) The requirements I added were from the experience I had when we used this patched LogFormatter to provide additional data needed to debug https://issues.apache.org/jira/browse/ZOOKEEPER-1208 . I dont think its ambitious, its based on whether you want to say its not a bug(I guess thats what you said in that 1208) because the user cant provide additional/relevant data or have the ability to ask user to run a command(which would have been a better thing) and give its output and you can reason about it. We had to spend quite some time to actually provide the relevant data needed for the ticket to actually find the issue and be solved. Don't know if its the right thing to expect every one to understand the transaction log to provide the relevant data.
        Hide
        Camille Fournier added a comment -

        Honestly, I think you're getting a bit ambitious for this ticket. I think you should simply enhance the logformatter to a degree that makes sense, and any additional tooling either make a new ticket or perhaps a github project for the work.

        Show
        Camille Fournier added a comment - Honestly, I think you're getting a bit ambitious for this ticket. I think you should simply enhance the logformatter to a degree that makes sense, and any additional tooling either make a new ticket or perhaps a github project for the work.
        Hide
        kishore gopalakrishna added a comment -

        Hi Camille,

        Yes, I wanted to enhance it as well. I was actually planning enhance it to a audit log and have a tool called zk data tree timeline.

        Requirements were

        • For a given path show the create, modify, delete sequence for a given range of time.
        • Given a session tell when was it created, who created it (unfortunately i cant get ip from transaction log), when was it closed.

        There were few more things I wanted but dont remember now.

        The Zk time line tool i had in mind was a applet with a time slider. It show the zk data tree. One should be able to use the slider to move back and forth in time and see the tree. This is very useful in debugging issues.

        I have written a tool(not ui) for my own project which parses the ZK transaction log and actually gives us the ability to know sequence of events that happened in the application. Unfortunately this is very specific to my application and others may not be able to use this.

        Let me know what you about a tool like this. If there is enough interest, I can spend time making it generic enough for others to use.

        Show
        kishore gopalakrishna added a comment - Hi Camille, Yes, I wanted to enhance it as well. I was actually planning enhance it to a audit log and have a tool called zk data tree timeline. Requirements were For a given path show the create, modify, delete sequence for a given range of time. Given a session tell when was it created, who created it (unfortunately i cant get ip from transaction log), when was it closed. There were few more things I wanted but dont remember now. The Zk time line tool i had in mind was a applet with a time slider. It show the zk data tree. One should be able to use the slider to move back and forth in time and see the tree. This is very useful in debugging issues. I have written a tool(not ui) for my own project which parses the ZK transaction log and actually gives us the ability to know sequence of events that happened in the application. Unfortunately this is very specific to my application and others may not be able to use this. Let me know what you about a tool like this. If there is enough interest, I can spend time making it generic enough for others to use.
        Hide
        Camille Fournier added a comment -

        Kishore, are you still interested in working on this? I'm thinking of enhancing the LogFormatter a bit more cleanly, debating whether to work on your patch or start from scratch.

        Show
        Camille Fournier added a comment - Kishore, are you still interested in working on this? I'm thinking of enhancing the LogFormatter a bit more cleanly, debating whether to work on your patch or start from scratch.
        Hide
        kishore gopalakrishna added a comment -

        New output format
        time:8/15/11 2:52:10 PM PDT session:0x131cf1a236f001e cxid:0x0 zxid:0x1189 type:closeSession path:/test version:10 data:48656C6C6F20576F726C6421

        Used HexBinaryAdapter to marshal data byte[]
        To get the byte[] from data one can do new HexBinaryAdapter().unmarshal("48656C6C6F20576F726C6421").

        Can some one tell me when will a MultiTxn happen. Currently that is not formatted using the above logic.

        Show
        kishore gopalakrishna added a comment - New output format time:8/15/11 2:52:10 PM PDT session:0x131cf1a236f001e cxid:0x0 zxid:0x1189 type:closeSession path:/test version:10 data:48656C6C6F20576F726C6421 Used HexBinaryAdapter to marshal data byte[] To get the byte[] from data one can do new HexBinaryAdapter().unmarshal("48656C6C6F20576F726C6421"). Can some one tell me when will a MultiTxn happen. Currently that is not formatted using the above logic.
        Hide
        kishore gopalakrishna added a comment -

        Patch to provide additional information about the transaction

        Show
        kishore gopalakrishna added a comment - Patch to provide additional information about the transaction

          People

          • Assignee:
            kishore gopalakrishna
            Reporter:
            kishore gopalakrishna
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development