Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.4.0
    • Fix Version/s: 3.6.0
    • Component/s: contrib
    • Labels:
    • Environment:

      GNU/Linux i386/i686/x64_84

    • Release Note:
      Hide
      1. Export/import capability of binary data
      2. Null data handing
      3. Export/import/dump/diff capability of subtree
      4. Efficient subtree handling
      5. Improved logging
      6. Improved testability with unittests
      7. Option to dump/export on file
      8. Fix to handle new state introduced in ZOOKEEPER-1108
      Show
      1. Export/import capability of binary data 2. Null data handing 3. Export/import/dump/diff capability of subtree 4. Efficient subtree handling 5. Improved logging 6. Improved testability with unittests 7. Option to dump/export on file 8. Fix to handle new state introduced in ZOOKEEPER-1108
    • Tags:
      zktreeutil

      Description

      ============================================
      zktreeutil - Zookeeper Tree Data Utility
      Author: Anirban Roy (r_anirban at yahoo.com)
      Organization: Yahoo Inc.
      ============================================

      zktreeutil program is intended to manage and manipulate zk-tree data quickly,
      efficiently and with ease. The utility operates on free-form ZK-tree and hence
      can be used for any cluster managed by Zookeeper. Here are the basic
      functionalities -

      EXPORT: The whole/partial ZK-tree is exported into a XML file. This helps in
      capturing a current snapshot of the data for backup/analysis. For a subtree
      export, one need to specify the path to the ZK-subtree with proper option.
      Since Zookeeper store binary data against znode, the data dumped on xml file
      is based64 encoded with an attribute "encode=true". Optionally one may specify
      not to encode data with --noencode option if data stored on zookeeper is
      guaranteed to be text data.

      IMPORT: The ZK (sub)tree can be imported from XML into ZK cluster. This helps in
      priming the new ZK cluster with static configuration. The import can be
      non-intrusive by making only additions and modifications in the existing data.
      One may optionally delete existing (sub)tree before importing the new data
      with --force option. The znodes which carries an attribute "encode=true" will be
      decoded and written to zookeeper.

      DIFF: Creates a diff between live ZK data vs data saved in XML file. Diff can
      ignore some ZK-tree branches (possibly dynamic data) on reading the optional
      ignore flag from XML file. Taking diff on a ZK-subtree achieved by providing
      path to ZK-subtree with diff command.

      UPDATE: Make the incremental changes into the live ZK-tree from saved XML,
      essentially after running the diff.

      DUMP: Dumps the ZK (sub)tree on the standard output device reading either from
      live ZK server or XML file.

      The exported ZK data into XML file can be shortened by only keeping the static
      ZK nodes which are required to prime an application. The dynamic zk nodes
      (created on-the-fly) can be ignored by setting a 'ignore' attribute at the root
      node of the dynamic subtree (see tests/zk_sample.xml), possibly deleting all
      inner ZK nodes under that. Once ignored, the whole subtree is ignored during
      DIFF, UPDATE and WRITE.

      Pre-requisites
      --------------
      1. Linux system with 2.6.X kernel.
      2. Zookeeper C client library (locally built at ../../c/.libs) >= 3.X.X
      3. Development build libraries (rpm packages):
      a. boost-devel >= 1.32.0
      b. libxml2-devel >= 2.6.26
      c. log4cxx-devel >= 0.9.7-7
      d. openssl-devel >= 0.9.7a
      e. cppunit >= 1.12.0-2

      Build instructions
      ------------------
      1. cd into this directory
      2. autoreconf -if
      3. ./configure # Configure the build env
      4. make # Build the tool
      5. make check # Run unit-tests
      6. ./src/zktreeutil --help # Usage help

      Testing and usage of zktreeutil
      --------------------------------
      1. Run Zookeeper server locally on port 2181
      2. export LD_LIBRARY_PATH=../../c/.libs/:/usr/local/lib/
      3. ./src/zktreeutil --help # show help
      4. ./src/zktreeutil --zookeeper=localhost:2181 --import --xmlfile=tests/zkdata_test.xml 2>/dev/null # import sample ZK tree
      5. ./src/zktreeutil --zookeeper=localhost:2181 --dump --path=/myapp/version-1.0 2>/dev/null # dump Zk subtree
      5. ./src/zktreeutil --zookeeper=localhost:2181 --dump --depth=3 2>/dev/null # dump Zk tree till certain depth
      6. ./src/zktreeutil --xmlfile=zkdata_test.xml -D 2>/dev/null # dump the xml data
      7. Change zkdata_test.xml with adding/deleting/chaging some nodes
      8. ./src/zktreeutil -z localhost:2181 -F -x zkdata_test.xml -p /myapp/version-1.0/configuration 2>/dev/null # take a diff of changes
      9. ./src/zktreeutil -z localhost:2181 -E --noencode 2>/dev/null > zk_sample2.xml # export the mofied ZK tree
      10. ./src/zktreeutil -z localhost:2181 -U -x zkdata_test.xml -p /myapp/version-1.0/distributions 2>/dev/null # update with incr. changes
      11. ./src/zktreeutil --zookeeper=localhost:2181 --import --force --xmlfile=zk_sample2.xml 2>/dev/null # re-prime the ZK tree

      For more details of usage, please see the unit tests. Hope this helps. Please
      reach out to me for any bugs, comments or suggestions.

      1. zktreeutil.patch
        189 kB
        Anirban Roy

        Activity

        Anirban Roy created issue -
        Anirban Roy made changes -
        Field Original Value New Value
        Attachment zktreeutil.patch [ 12498286 ]
        Anirban Roy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Mahadev konar made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Anirban Roy made changes -
        Release Note 1. Export/import capability of binary data
        2. Null data handing
        3. Export/import/dump/diff capability of subtree
        4. Efficient subtree handling
        1. Export/import capability of binary data
        2. Null data handing
        3. Export/import/dump/diff capability of subtree
        4. Efficient subtree handling
        5. Improved logging
        6. Improved testability with unittests
        7. Option to dump/export on file
        8. Fix to handle new state introduced in ZOOKEEPER-1108
        Fix Version/s 3.4.0 [ 12314469 ]
        Fix Version/s 3.5.0 [ 12316644 ]
        Affects Version/s 3.4.0 [ 12314469 ]
        Affects Version/s 3.3.3 [ 12315482 ]
        Description ==========================================
        zktreeutil - Zookeeper Tree Data Utility
        Author: Anirban Roy (r_anirban@yahoo.com)
        Organization: Yahoo Inc.
        ==========================================

        zktreeutil program is intended to manage and manipulate zk-tree data quickly,
        efficiently and with ease. The utility operates on free-form ZK-tree and hence
        can be used for any cluster managed by Zookeeper. Here are the basic
        functionalities -

        EXPORT: The whole/partial ZK-tree is exported into a XML file. This helps in
        capturing a current snapshot of the data for backup/analysis. For a subtree
        export, one need to specify the path to the ZK-subtree with proper option.
        Since Zookeeper store binary data against znode, the data dumped on xml file
        is based64 encoded with an attribute "encode=true". Optionally one may specify
        not to encode data with --noencode option if data stored on zookeeper is
        guaranteed to be text data.

        IMPORT: The ZK (sub)tree can be imported from XML into ZK cluster. This helps in
        priming the new ZK cluster with static configuration. The import can be
        non-intrusive by making only additions and modifications in the existing data.
        One may optionally delete existing (sub)tree before importing the new data
        with --force option. The znodes which carries an attribute "encode=true" will be
        decoded and written to zookeeper.

        DIFF: Creates a diff between live ZK data vs data saved in XML file. Diff can
        ignore some ZK-tree branches (possibly dynamic data) on reading the optional
        ignore flag from XML file. Taking diff on a ZK-subtree achieved by providing
        path to ZK-subtree with diff command.

        UPDATE: Make the incremental changes into the live ZK-tree from saved XML,
        essentially after running the diff.

        DUMP: Dumps the ZK (sub)tree on the standard output device reading either from
        live ZK server or XML file.

        The exported ZK data into XML file can be shortened by only keeping the static
        ZK nodes which are required to prime an application. The dynamic zk nodes
        (created on-the-fly) can be ignored by setting a 'ignore' attribute at the root
        node of the dynamic subtree (see tests/zk_sample.xml), possibly deleting all
        inner ZK nodes under that. Once ignored, the whole subtree is ignored during
        DIFF, UPDATE and WRITE.

        Pre-requisites
        --------------
        1. Linux system with 2.6.X kernel.
        2. Zookeeper C client library (locally built at ../../c/.libs) >= 3.X.X
        3. Development build libraries (rpm packages):
          a. boost-devel >= 1.32.0
          b. libxml2-devel >= 2.7.3
          c. log4cxx0100-devel >= 0.9.0
          d. openssl-devel >= 0.9.7a

        Build instructions
        ------------------
        1. cd into this directory
        2. autoreconf -if
        3. ./configure
        4. make
        5. 'zktreeutil' binary created under src directory

        Testing and usage of zktreeutil
        --------------------------------
        1. Run Zookeeper server locally on port 2181
        2. export LD_LIBRARY_PATH=../../c/.libs/:/usr/local/lib/
        3. ./src/zktreeutil --help # show help
        4. ./src/zktreeutil --zookeeper=localhost:2181 --import --xmlfile=tests/zk_sample.xml 2>/dev/null # import sample ZK tree
        5. ./src/zktreeutil --zookeeper=localhost:2181 --dump --path=/myapp/version-1.0 2>/dev/null # dump Zk subtree
        6. ./src/zktreeutil --zookeeper=localhost:2181 --dump --depth=3 2>/dev/null # dump Zk tree till certain depth
        7. ./src/zktreeutil --xmlfile=zk_sample.xml -D 2>/dev/null # dump the xml data
        8. Change zk_sample.xml with adding/deleting/chaging some nodes
        9. ./src/zktreeutil -z localhost:2181 -F -x zk_sample.xml -p /myapp/version-1.0/configuration 2>/dev/null # take a diff of changes
        10. ./src/zktreeutil -z localhost:2181 -E --noencode 2>/dev/null > zk_sample2.xml # export the mofied ZK tree
        11. ./src/zktreeutil -z localhost:2181 -U -x zk_sample.xml -p /myapp/version-1.0/distributions 2>/dev/null # update with incr. changes
        12. ./src/zktreeutil --zookeeper=localhost:2181 --import --force --xmlfile=zk_sample2.xml 2>/dev/null # re-prime the ZK tree
        ============================================
        zktreeutil - Zookeeper Tree Data Utility
        Author: Anirban Roy (r_anirban at yahoo.com)
        Organization: Yahoo Inc.
        ============================================

        zktreeutil program is intended to manage and manipulate zk-tree data quickly,
        efficiently and with ease. The utility operates on free-form ZK-tree and hence
        can be used for any cluster managed by Zookeeper. Here are the basic
        functionalities -

        EXPORT: The whole/partial ZK-tree is exported into a XML file. This helps in
        capturing a current snapshot of the data for backup/analysis. For a subtree
        export, one need to specify the path to the ZK-subtree with proper option.
        Since Zookeeper store binary data against znode, the data dumped on xml file
        is based64 encoded with an attribute "encode=true". Optionally one may specify
        not to encode data with --noencode option if data stored on zookeeper is
        guaranteed to be text data.

        IMPORT: The ZK (sub)tree can be imported from XML into ZK cluster. This helps in
        priming the new ZK cluster with static configuration. The import can be
        non-intrusive by making only additions and modifications in the existing data.
        One may optionally delete existing (sub)tree before importing the new data
        with --force option. The znodes which carries an attribute "encode=true" will be
        decoded and written to zookeeper.

        DIFF: Creates a diff between live ZK data vs data saved in XML file. Diff can
        ignore some ZK-tree branches (possibly dynamic data) on reading the optional
        ignore flag from XML file. Taking diff on a ZK-subtree achieved by providing
        path to ZK-subtree with diff command.

        UPDATE: Make the incremental changes into the live ZK-tree from saved XML,
        essentially after running the diff.

        DUMP: Dumps the ZK (sub)tree on the standard output device reading either from
        live ZK server or XML file.

        The exported ZK data into XML file can be shortened by only keeping the static
        ZK nodes which are required to prime an application. The dynamic zk nodes
        (created on-the-fly) can be ignored by setting a 'ignore' attribute at the root
        node of the dynamic subtree (see tests/zk_sample.xml), possibly deleting all
        inner ZK nodes under that. Once ignored, the whole subtree is ignored during
        DIFF, UPDATE and WRITE.

        Pre-requisites
        --------------
        1. Linux system with 2.6.X kernel.
        2. Zookeeper C client library (locally built at ../../c/.libs) >= 3.X.X
        3. Development build libraries (rpm packages):
          a. boost-devel >= 1.32.0
          b. libxml2-devel >= 2.6.26
          c. log4cxx-devel >= 0.9.7-7
          d. openssl-devel >= 0.9.7a
          e. cppunit >= 1.12.0-2

        Build instructions
        ------------------
        1. cd into this directory
        2. autoreconf -if
        3. ./configure # Configure the build env
        4. make # Build the tool
        5. make check # Run unit-tests
        6. ./src/zktreeutil --help # Usage help

        Testing and usage of zktreeutil
        --------------------------------
        1. Run Zookeeper server locally on port 2181
        2. export LD_LIBRARY_PATH=../../c/.libs/:/usr/local/lib/
        3. ./src/zktreeutil --help # show help
        4. ./src/zktreeutil --zookeeper=localhost:2181 --import --xmlfile=tests/zkdata_test.xml 2>/dev/null # import sample ZK tree
        5. ./src/zktreeutil --zookeeper=localhost:2181 --dump --path=/myapp/version-1.0 2>/dev/null # dump Zk subtree
        5. ./src/zktreeutil --zookeeper=localhost:2181 --dump --depth=3 2>/dev/null # dump Zk tree till certain depth
        6. ./src/zktreeutil --xmlfile=zkdata_test.xml -D 2>/dev/null # dump the xml data
        7. Change zkdata_test.xml with adding/deleting/chaging some nodes
        8. ./src/zktreeutil -z localhost:2181 -F -x zkdata_test.xml -p /myapp/version-1.0/configuration 2>/dev/null # take a diff of changes
        9. ./src/zktreeutil -z localhost:2181 -E --noencode 2>/dev/null > zk_sample2.xml # export the mofied ZK tree
        10. ./src/zktreeutil -z localhost:2181 -U -x zkdata_test.xml -p /myapp/version-1.0/distributions 2>/dev/null # update with incr. changes
        11. ./src/zktreeutil --zookeeper=localhost:2181 --import --force --xmlfile=zk_sample2.xml 2>/dev/null # re-prime the ZK tree

        For more details of usage, please see the unit tests. Hope this helps. Please
        reach out to me for any bugs, comments or suggestions.
        Anirban Roy made changes -
        Attachment zktreeutil.patch [ 12522767 ]
        Anirban Roy made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Anirban Roy made changes -
        Status In Progress [ 3 ] Open [ 1 ]
        Anirban Roy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Anirban Roy made changes -
        Attachment zktreeutil.patch [ 12522937 ]
        Anirban Roy made changes -
        Attachment zktreeutil.patch [ 12522940 ]
        Flavio Junqueira made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Anirban Roy made changes -
        Attachment zktreeutil.patch [ 12498286 ]
        Anirban Roy made changes -
        Attachment zktreeutil.patch [ 12522767 ]
        Anirban Roy made changes -
        Attachment zktreeutil.patch [ 12522937 ]
        Anirban Roy made changes -
        Attachment zktreeutil.patch [ 12522940 ]
        Anirban Roy made changes -
        Attachment zktreeutil.patch [ 12522983 ]
        Anirban Roy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Anirban Roy made changes -
        Attachment zktreeutil.patch [ 12522983 ]
        Anirban Roy made changes -
        Attachment zktreeutil.patch [ 12523179 ]
        Anirban Roy made changes -
        Remaining Estimate 168h [ 604800 ] 0h [ 0 ]
        Time Spent 168h [ 604800 ]
        Worklog Id 13334 [ 13334 ]
        Mahadev konar made changes -
        Fix Version/s 3.5.0 [ 12316644 ]
        Fix Version/s 3.4.0 [ 12314469 ]
        Michi Mutsuzaki made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Michi Mutsuzaki made changes -
        Fix Version/s 3.6.0 [ 12326518 ]
        Fix Version/s 3.5.0 [ 12316644 ]

          People

          • Assignee:
            Anirban Roy
            Reporter:
            Anirban Roy
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 168h
              168h
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 168h
              168h

                Development