Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.4.0
    • Fix Version/s: 3.5.0
    • Component/s: contrib
    • Labels:
    • Environment:

      GNU/Linux i386/i686/x64_84

    • Release Note:
      Hide
      1. Export/import capability of binary data
      2. Null data handing
      3. Export/import/dump/diff capability of subtree
      4. Efficient subtree handling
      5. Improved logging
      6. Improved testability with unittests
      7. Option to dump/export on file
      8. Fix to handle new state introduced in ZOOKEEPER-1108
      Show
      1. Export/import capability of binary data 2. Null data handing 3. Export/import/dump/diff capability of subtree 4. Efficient subtree handling 5. Improved logging 6. Improved testability with unittests 7. Option to dump/export on file 8. Fix to handle new state introduced in ZOOKEEPER-1108
    • Tags:
      zktreeutil

      Description

      ============================================
      zktreeutil - Zookeeper Tree Data Utility
      Author: Anirban Roy (r_anirban at yahoo.com)
      Organization: Yahoo Inc.
      ============================================

      zktreeutil program is intended to manage and manipulate zk-tree data quickly,
      efficiently and with ease. The utility operates on free-form ZK-tree and hence
      can be used for any cluster managed by Zookeeper. Here are the basic
      functionalities -

      EXPORT: The whole/partial ZK-tree is exported into a XML file. This helps in
      capturing a current snapshot of the data for backup/analysis. For a subtree
      export, one need to specify the path to the ZK-subtree with proper option.
      Since Zookeeper store binary data against znode, the data dumped on xml file
      is based64 encoded with an attribute "encode=true". Optionally one may specify
      not to encode data with --noencode option if data stored on zookeeper is
      guaranteed to be text data.

      IMPORT: The ZK (sub)tree can be imported from XML into ZK cluster. This helps in
      priming the new ZK cluster with static configuration. The import can be
      non-intrusive by making only additions and modifications in the existing data.
      One may optionally delete existing (sub)tree before importing the new data
      with --force option. The znodes which carries an attribute "encode=true" will be
      decoded and written to zookeeper.

      DIFF: Creates a diff between live ZK data vs data saved in XML file. Diff can
      ignore some ZK-tree branches (possibly dynamic data) on reading the optional
      ignore flag from XML file. Taking diff on a ZK-subtree achieved by providing
      path to ZK-subtree with diff command.

      UPDATE: Make the incremental changes into the live ZK-tree from saved XML,
      essentially after running the diff.

      DUMP: Dumps the ZK (sub)tree on the standard output device reading either from
      live ZK server or XML file.

      The exported ZK data into XML file can be shortened by only keeping the static
      ZK nodes which are required to prime an application. The dynamic zk nodes
      (created on-the-fly) can be ignored by setting a 'ignore' attribute at the root
      node of the dynamic subtree (see tests/zk_sample.xml), possibly deleting all
      inner ZK nodes under that. Once ignored, the whole subtree is ignored during
      DIFF, UPDATE and WRITE.

      Pre-requisites
      --------------
      1. Linux system with 2.6.X kernel.
      2. Zookeeper C client library (locally built at ../../c/.libs) >= 3.X.X
      3. Development build libraries (rpm packages):
      a. boost-devel >= 1.32.0
      b. libxml2-devel >= 2.6.26
      c. log4cxx-devel >= 0.9.7-7
      d. openssl-devel >= 0.9.7a
      e. cppunit >= 1.12.0-2

      Build instructions
      ------------------
      1. cd into this directory
      2. autoreconf -if
      3. ./configure # Configure the build env
      4. make # Build the tool
      5. make check # Run unit-tests
      6. ./src/zktreeutil --help # Usage help

      Testing and usage of zktreeutil
      --------------------------------
      1. Run Zookeeper server locally on port 2181
      2. export LD_LIBRARY_PATH=../../c/.libs/:/usr/local/lib/
      3. ./src/zktreeutil --help # show help
      4. ./src/zktreeutil --zookeeper=localhost:2181 --import --xmlfile=tests/zkdata_test.xml 2>/dev/null # import sample ZK tree
      5. ./src/zktreeutil --zookeeper=localhost:2181 --dump --path=/myapp/version-1.0 2>/dev/null # dump Zk subtree
      5. ./src/zktreeutil --zookeeper=localhost:2181 --dump --depth=3 2>/dev/null # dump Zk tree till certain depth
      6. ./src/zktreeutil --xmlfile=zkdata_test.xml -D 2>/dev/null # dump the xml data
      7. Change zkdata_test.xml with adding/deleting/chaging some nodes
      8. ./src/zktreeutil -z localhost:2181 -F -x zkdata_test.xml -p /myapp/version-1.0/configuration 2>/dev/null # take a diff of changes
      9. ./src/zktreeutil -z localhost:2181 -E --noencode 2>/dev/null > zk_sample2.xml # export the mofied ZK tree
      10. ./src/zktreeutil -z localhost:2181 -U -x zkdata_test.xml -p /myapp/version-1.0/distributions 2>/dev/null # update with incr. changes
      11. ./src/zktreeutil --zookeeper=localhost:2181 --import --force --xmlfile=zk_sample2.xml 2>/dev/null # re-prime the ZK tree

      For more details of usage, please see the unit tests. Hope this helps. Please
      reach out to me for any bugs, comments or suggestions.

      1. zktreeutil.patch
        189 kB
        Anirban Roy

        Activity

        Hide
        Anirban Roy added a comment -

        Submitted patch.

        Show
        Anirban Roy added a comment - Submitted patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12498286/zktreeutil.patch
        against trunk revision 1180217.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/614//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/614//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/614//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498286/zktreeutil.patch against trunk revision 1180217. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/614//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/614//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/614//console This message is automatically generated.
        Hide
        Mahadev konar added a comment -

        Anirban,
        I'd really like to have a discussion on where this code should go. Adding contribs seems to be creating more and more burden on ZK devs. Can you please start a thread on dev list regarding addition of this utility as contrib? This way we can all agree where this should go long term and where we the best place to host this code shorter term.

        Show
        Mahadev konar added a comment - Anirban, I'd really like to have a discussion on where this code should go. Adding contribs seems to be creating more and more burden on ZK devs. Can you please start a thread on dev list regarding addition of this utility as contrib? This way we can all agree where this should go long term and where we the best place to host this code shorter term.
        Hide
        Anirban Roy added a comment -

        Hi Mahadev, this utility is already available as contrib, I just enhanced the utility and put a patch to apply. We are using this utility heavily and want it to be part of zookeeper code with all the fixes and enhancements. Please let me know if I still need to start a thread on the location of this piece.

        Show
        Anirban Roy added a comment - Hi Mahadev, this utility is already available as contrib, I just enhanced the utility and put a patch to apply. We are using this utility heavily and want it to be part of zookeeper code with all the fixes and enhancements. Please let me know if I still need to start a thread on the location of this piece.
        Hide
        Jason Hane added a comment -

        Using the original zktreeutil and this patched version, I always get an error trying to import, update or diff the tree. Export and dump are the only features that work.

        [zktreeutil] unknown znode during traversal:
        Aborted

        I added some debugging to the traverseBranch_ method of ZkTreeUtil.cc and it turns out during the if statement comparison of the childNodeSptr->getKey() and nodes[znode_idx], nodes[znode_idx] is empty. In fact I don't think nodes is populated at all.

        If there is anything I can give you to help replicate the problem please let me know. This tool is vital to our needs but we are unable to use it.

        Show
        Jason Hane added a comment - Using the original zktreeutil and this patched version, I always get an error trying to import, update or diff the tree. Export and dump are the only features that work. [zktreeutil] unknown znode during traversal: Aborted I added some debugging to the traverseBranch_ method of ZkTreeUtil.cc and it turns out during the if statement comparison of the childNodeSptr->getKey() and nodes [znode_idx] , nodes [znode_idx] is empty. In fact I don't think nodes is populated at all. If there is anything I can give you to help replicate the problem please let me know. This tool is vital to our needs but we are unable to use it.
        Hide
        Anirban Roy added a comment -

        Hi Jason, we are using this patch in its full capability for quite some time. It may be possible that you are hitting some rare issue which we never faced. Can you please reproduce this and post here? I'm planning to add some unit tests along with the patch. Once I'm able to reproduce, can give you a fix quickly.

        Show
        Anirban Roy added a comment - Hi Jason, we are using this patch in its full capability for quite some time. It may be possible that you are hitting some rare issue which we never faced. Can you please reproduce this and post here? I'm planning to add some unit tests along with the patch. Once I'm able to reproduce, can give you a fix quickly.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522767/zktreeutil.patch
        against trunk revision 1326029.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 18 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1035//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522767/zktreeutil.patch against trunk revision 1326029. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1035//console This message is automatically generated.
        Hide
        Anirban Roy added a comment -

        Zktreeutil patch

        Show
        Anirban Roy added a comment - Zktreeutil patch
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522937/zktreeutil.patch
        against trunk revision 1326029.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 18 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1036//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522937/zktreeutil.patch against trunk revision 1326029. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1036//console This message is automatically generated.
        Hide
        Flavio Junqueira added a comment -

        hi anirban, you need to also to grant the license to ASF when you submit the patch. otherwise we can't include it, even if the patch is good.

        Show
        Flavio Junqueira added a comment - hi anirban, you need to also to grant the license to ASF when you submit the patch. otherwise we can't include it, even if the patch is good.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522940/zktreeutil.patch
        against trunk revision 1326029.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 51 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1037//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1037//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1037//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522940/zktreeutil.patch against trunk revision 1326029. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 51 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1037//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1037//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1037//console This message is automatically generated.
        Hide
        Anirban Roy added a comment -

        zktreeutil patch with ASF license.

        Show
        Anirban Roy added a comment - zktreeutil patch with ASF license.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522983/zktreeutil.patch
        against trunk revision 1326029.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 51 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1038//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1038//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1038//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522983/zktreeutil.patch against trunk revision 1326029. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 51 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1038//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1038//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1038//console This message is automatically generated.
        Hide
        Anirban Roy added a comment -

        Patch with removed boost dependency. Different boost versions in i386 and x86 arch was preventing the codebase to be portable.

        Show
        Anirban Roy added a comment - Patch with removed boost dependency. Different boost versions in i386 and x86 arch was preventing the codebase to be portable.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12523179/zktreeutil.patch
        against trunk revision 1326029.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 51 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1040//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1040//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1040//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523179/zktreeutil.patch against trunk revision 1326029. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 51 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1040//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1040//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1040//console This message is automatically generated.
        Hide
        Michi Mutsuzaki added a comment -

        Let's address Mahadev's comment before moving forward with this patch.

        Show
        Michi Mutsuzaki added a comment - Let's address Mahadev's comment before moving forward with this patch.

          People

          • Assignee:
            Anirban Roy
            Reporter:
            Anirban Roy
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 168h
              168h
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 168h
              168h

                Development