Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      I've put together a basic component implementation for interacting with a Zookeeper Cluster (http://hadoop.apache.org/zookeeper/). Its a reasonably complete implementation, providing much of the available functionality of the 3.3.0 api. The main features being the abilities to

      • Creation of nodes in any of the ZooKeeper create modes.
      • Get and Set the data contents of arbitrary cluster nodes.
      • Create and retrieve the list the child nodes attached to a particular node.
      • A Distributed RoutePoilcy that leverages a Leader election coordinated by ZK to determine if exchanges should get processed.

      It's build against the Head of Zookeepers current release stream, 3.3.0. Zookeeper (Like the rest of the Hadoop project) uses a custom ant build and so there are no SNAPSHOTS. This is a bit of an impediment, but its likely that the Zookeepr project will cut a release before too long. In the meantime, to get some soak time in the real world, I've attached a freshly cut jar from the head today so you can try the component out. It can be installed using the following command.

      mvn install:install-file -DgroupId=org.apache.zookeeper -DartifactId=zookeeper -Dversion=3.3.0 -Dpackaging=jar -Dfile=zookeeper-3.3.0.jar

      I'm putting together some documentation and will be happy to add it to the wiki if the component gets picked up. I hope the project can find this useful.

      1. zookeeper-3.3.0.jar
        1.27 MB
        Stephen Gargan
      2. zookeeper.patch
        176 kB
        Stephen Gargan
      3. zookeeper.patch
        180 kB
        Stephen Gargan

        Activity

        Hide
        Gert Vanthienen added a comment -

        We could always add this artifact to http://svn.apache.org/repos/asf/camel/m2-repo/ and rename it 3.3.0-SNAPSHOT until the Zookeeper release has been cut (or create a m2-snapshots repo to avoid mixing snapshot and releases in our existing repo).

        Show
        Gert Vanthienen added a comment - We could always add this artifact to http://svn.apache.org/repos/asf/camel/m2-repo/ and rename it 3.3.0-SNAPSHOT until the Zookeeper release has been cut (or create a m2-snapshots repo to avoid mixing snapshot and releases in our existing repo).
        Hide
        Claus Ibsen added a comment -

        Just to be clear the OS license for ZooKeeper is that Apache friendly?

        Show
        Claus Ibsen added a comment - Just to be clear the OS license for ZooKeeper is that Apache friendly?
        Hide
        Stephen Gargan added a comment -

        Yes absolutely. Zookeeper is a sub project of Hadoop, a top level
        Apache project. The license is Apache 2.0

        http://svn.apache.org/repos/asf/hadoop/zookeeper/trunk/LICENSE.txt

        Show
        Stephen Gargan added a comment - Yes absolutely. Zookeeper is a sub project of Hadoop, a top level Apache project. The license is Apache 2.0 http://svn.apache.org/repos/asf/hadoop/zookeeper/trunk/LICENSE.txt
        Hide
        Willem Jiang added a comment -

        Another question, which version of JDK does Zookeeper need to run with?
        We can't put the camel hadoop HFS component into Apache camel as it need to run with JDK 1.6.

        Show
        Willem Jiang added a comment - Another question, which version of JDK does Zookeeper need to run with? We can't put the camel hadoop HFS component into Apache camel as it need to run with JDK 1.6.
        Hide
        Claus Ibsen added a comment -

        @Willem yeah that was a good point. We had to host the other hadoop project at camel-extra since it uses JDK1.6+

        See more here:
        http://camel.apache.org/hdfs.html

        Show
        Claus Ibsen added a comment - @Willem yeah that was a good point. We had to host the other hadoop project at camel-extra since it uses JDK1.6+ See more here: http://camel.apache.org/hdfs.html
        Hide
        Stephen Gargan added a comment -

        I've been running it with 1.6. but it builds with 1.5 as the target.

        http://svn.apache.org/repos/asf/hadoop/zookeeper/trunk/build.xml

        Show
        Stephen Gargan added a comment - I've been running it with 1.6. but it builds with 1.5 as the target. http://svn.apache.org/repos/asf/hadoop/zookeeper/trunk/build.xml
        Hide
        Stephen Gargan added a comment -

        So looking a little harder, the Zookeeper docs do specify 1.6 as a requirement. http://hadoop.apache.org/zookeeper/docs/r3.2.2/zookeeperAdmin.html. I had a fun ol' time trying to get a 1.5 JDK from the Sun/Oracle site now that it has reached its end of life. I will test tomorrow and see if Zookeeper will run with it but i guess its to 'extras' for now if at all.

        Any plans for a 1.6 migration or are there too many customers invested in 1.5?

        Show
        Stephen Gargan added a comment - So looking a little harder, the Zookeeper docs do specify 1.6 as a requirement. http://hadoop.apache.org/zookeeper/docs/r3.2.2/zookeeperAdmin.html . I had a fun ol' time trying to get a 1.5 JDK from the Sun/Oracle site now that it has reached its end of life. I will test tomorrow and see if Zookeeper will run with it but i guess its to 'extras' for now if at all. Any plans for a 1.6 migration or are there too many customers invested in 1.5?
        Hide
        Claus Ibsen added a comment -

        Yeah lets host it at camel-extra.

        When eventually we can migrate Camel to 1.6 as min, then we can move some of the components from camel-extra to Apache

        Show
        Claus Ibsen added a comment - Yeah lets host it at camel-extra. When eventually we can migrate Camel to 1.6 as min, then we can move some of the components from camel-extra to Apache
        Hide
        Stephen Gargan added a comment -

        I've added a repo to my account at github.com to host this for the moment http://github.com/sgargan/camel-zookeeper. Hopefully at some stage it can make it into the mainline proper, but for now its good where it is. In the meantime, should I add documentation to the wiki or hold off?

        Show
        Stephen Gargan added a comment - I've added a repo to my account at github.com to host this for the moment http://github.com/sgargan/camel-zookeeper . Hopefully at some stage it can make it into the mainline proper, but for now its good where it is. In the meantime, should I add documentation to the wiki or hold off?
        Hide
        Hadrian Zbarcea added a comment -

        Stephen, are you still interested in donating this component?

        Camel has now java6 as a minimum requirement. I saw that your code at github uses 2.7-SNAPSHOT, it'll require minor changes and we could have it in 2.9.0 (2.8.0 is currently under vote).

        Show
        Hadrian Zbarcea added a comment - Stephen, are you still interested in donating this component? Camel has now java6 as a minimum requirement. I saw that your code at github uses 2.7-SNAPSHOT, it'll require minor changes and we could have it in 2.9.0 (2.8.0 is currently under vote).
        Hide
        Stephen Gargan added a comment -

        Hadrian,

        Absolutely! One minor hiccup is that I'm on the road backpacking at the
        moment and am laptopless. I should be back home in 3 weeks and if it can
        wait till then I will gladly fix it up for inclusion.

        Thanks

        Steve.

        Show
        Stephen Gargan added a comment - Hadrian, Absolutely! One minor hiccup is that I'm on the road backpacking at the moment and am laptopless. I should be back home in 3 weeks and if it can wait till then I will gladly fix it up for inclusion. Thanks Steve.
        Hide
        Hadrian Zbarcea added a comment -

        Stephen, thx, no problem. There's plenty of time to get it done for 2.9.0 (maybe even a 2.8.x patch).

        Show
        Hadrian Zbarcea added a comment - Stephen, thx, no problem. There's plenty of time to get it done for 2.9.0 (maybe even a 2.8.x patch).
        Hide
        Stephen Gargan added a comment -

        I've updated the component to work against trunk and attached an updated patch. I've put together some basic documentation for it here https://cwiki.apache.org/confluence/display/CAMEL/Zookeeper. The components wiki page will need an entry eventually.

        Let me know if there is anything else needed.

        Show
        Stephen Gargan added a comment - I've updated the component to work against trunk and attached an updated patch. I've put together some basic documentation for it here https://cwiki.apache.org/confluence/display/CAMEL/Zookeeper . The components wiki page will need an entry eventually. Let me know if there is anything else needed.
        Hide
        Hadrian Zbarcea added a comment -

        @Stephen, thanks a lot for the patch and the documentation. I started to review it. At first glance it looks great. Given its size I am not sure if it would require a code grant, probably not, but I'll have to figure it out. Will let you know either way in a day or two.

        Show
        Hadrian Zbarcea added a comment - @Stephen, thanks a lot for the patch and the documentation. I started to review it. At first glance it looks great. Given its size I am not sure if it would require a code grant, probably not, but I'll have to figure it out. Will let you know either way in a day or two.
        Hide
        Hadrian Zbarcea added a comment -

        Patch applied in r1161306 with a minor change accounting for cschneider's refactoring and one failing test I am looking into. There are a few other minor issues I will look into related to forkmode=pertest, that shouldn't be necessary, Tread.sleeps etc. I'll get those fixed and close this issue. The documentation is already done. Excellent contribution Stephen, much appreciated.

        Show
        Hadrian Zbarcea added a comment - Patch applied in r1161306 with a minor change accounting for cschneider's refactoring and one failing test I am looking into. There are a few other minor issues I will look into related to forkmode=pertest, that shouldn't be necessary, Tread.sleeps etc. I'll get those fixed and close this issue. The documentation is already done. Excellent contribution Stephen, much appreciated.
        Hide
        Claus Ibsen added a comment - - edited

        Thanks Stephen for this new great component.

        I have polished the code a bit to use slf4j logging as we use in Camel by default. I also removed some not needed dependency in the pom.xml such as on camel-spring. As well polished the unit tests to not depend on camel-core-test JAR as this is discouraged.
        As well as making the packaging as an osgi bundle.

        I would suggest to only keep the zookeeper as the name for the component in the DSL. I am not really a fan of "zoo".
        Anyone object to remove "zoo" so you would have to use "zookeeper" in the DSL?

        Show
        Claus Ibsen added a comment - - edited Thanks Stephen for this new great component. I have polished the code a bit to use slf4j logging as we use in Camel by default. I also removed some not needed dependency in the pom.xml such as on camel-spring. As well polished the unit tests to not depend on camel-core-test JAR as this is discouraged. As well as making the packaging as an osgi bundle. I would suggest to only keep the zookeeper as the name for the component in the DSL. I am not really a fan of "zoo". Anyone object to remove "zoo" so you would have to use "zookeeper" in the DSL?
        Hide
        Stephen Gargan added a comment -

        Very welcome, glad to help.

        Show
        Stephen Gargan added a comment - Very welcome, glad to help.

          People

          • Assignee:
            Hadrian Zbarcea
            Reporter:
            Stephen Gargan
          • Votes:
            2 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development