Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.4.0
    • Component/s: contrib
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I would like to extend the existing REST Interface to also support:

      • configuration
      • ephemeral znodes
      • watches - PubSubHubbub
      • ACLs
      • basic authentication

      I want to do this because when building web applications that talks directly to ZooKeeper a REST API it's a lot easier to use (there is no protocol mismatch) than an API that uses persistent connections. I plan to use the improved version to build a web-based administrative interface.

      1. keys.tar.gz
        2 kB
        Andrei Savu
      2. SPEC.txt
        13 kB
        Andrei Savu
      3. SPEC.txt
        14 kB
        Andrei Savu
      4. ZOOKEEPER-809.patch
        112 kB
        Andrei Savu
      5. ZOOKEEPER-809.patch
        112 kB
        Andrei Savu
      6. ZOOKEEPER-809.patch
        109 kB
        Andrei Savu
      7. ZOOKEEPER-809.patch
        99 kB
        Andrei Savu
      8. ZOOKEEPER-809.patch
        96 kB
        Andrei Savu
      9. ZOOKEEPER-809.patch
        67 kB
        Andrei Savu

        Issue Links

          Activity

          Andrei Savu created issue -
          Hide
          Andrei Savu added a comment -

          Very soon I'm also going to attach an updated version of the REST SPEC.txt file because I would like to get more feedback from you before writing any code.

          Show
          Andrei Savu added a comment - Very soon I'm also going to attach an updated version of the REST SPEC.txt file because I would like to get more feedback from you before writing any code.
          Andrei Savu made changes -
          Field Original Value New Value
          Link This issue is required by ZOOKEEPER-701 [ ZOOKEEPER-701 ]
          Andrei Savu made changes -
          Link This issue is required by ZOOKEEPER-808 [ ZOOKEEPER-808 ]
          Hide
          Andrei Savu added a comment -

          I've attached the first draft of the SPECs for an improved REST gateway.

          I'm proposing the following new operations:

          1. create a new session – POST /sessions/v1?op=create HTTP/1.1

          2. keep the session alive – PUT /sessions/v1/<SESSION-ID> HTTP/1.1

          3. close the session – DELETE /sessions/v1/<SESSION-ID> HTTP/1.1

          4. create an ephemeral node – POST /znodes/v1/a/b?op=create&name=c&ephemeral=true&session=<SESSION-ID> HTTP/1.1

          5. create a new watch – POST /znodes/v1/a/b?op=watch&view=data OR children&session=<SESSION-ID> HTTP/1.1

          6. query watch status – GET /sessions/v1/<SESSION-ID>/watches/<WATCH-ID> HTTP/1.1
          This operation could support long-polling in the future.

          7. delete a triggered watch – DELETE /sessions/v1/<SESSION-ID>/watches/<WATCH-ID> HTTP/1.1

          Let me know what you think about this. Am I breaking the REST principles?

          This is what I want to do in the first iteration. In the second iteration I would like to add support for ACLs and authentication.

          Show
          Andrei Savu added a comment - I've attached the first draft of the SPECs for an improved REST gateway. I'm proposing the following new operations: 1. create a new session – POST /sessions/v1?op=create HTTP/1.1 2. keep the session alive – PUT /sessions/v1/<SESSION-ID> HTTP/1.1 3. close the session – DELETE /sessions/v1/<SESSION-ID> HTTP/1.1 4. create an ephemeral node – POST /znodes/v1/a/b?op=create&name=c&ephemeral=true&session=<SESSION-ID> HTTP/1.1 5. create a new watch – POST /znodes/v1/a/b?op=watch&view=data OR children&session=<SESSION-ID> HTTP/1.1 6. query watch status – GET /sessions/v1/<SESSION-ID>/watches/<WATCH-ID> HTTP/1.1 This operation could support long-polling in the future. 7. delete a triggered watch – DELETE /sessions/v1/<SESSION-ID>/watches/<WATCH-ID> HTTP/1.1 Let me know what you think about this. Am I breaking the REST principles? This is what I want to do in the first iteration. In the second iteration I would like to add support for ACLs and authentication.
          Andrei Savu made changes -
          Attachment SPEC.txt [ 12449455 ]
          Hide
          Patrick Hunt added a comment -

          When I first created the REST interface I didn't have the notion of sessions, now that you do I think you would want to augment the notion of having a /znodes/... url with a url of /sessions/v1/<session TOKEN>/znodes/....

          so create the session as you suggest, however that create operation returns a url representing the session, after which all of your operations use that as a "prefix" if you will. e.g.:

          create a new session - POST /sessions/v1?op=create HTTP/1.1

          returns /sessions/v1/ab483cd8283ef274

          notice the session TOKEN is a randomly generated key - this allows for some "security through obscurity" as it's "hard to guess" and is some small measure of security. session keepalive and delete would operate on this url. GET on the url might return the original session id for example

          create an ephemeral node - POST /sessions/v1/ab483cd8283ef274/znodes/v1/a/b?op=create&name=c&ephemeral=true HTTP/1.1

          you might keep the /znodes feature as-is for those not wanting to use sessions (admin r/o console say, or cli tool), however you might want to make turning it off an option - allowing the operator to force users to use explicit sessions

          notice how this also cleans up items 5/6 wrt the url used to access (same prefix in both cases)

          when you add acl support you might add something like:

          /sessions/v1/ab483cd8283ef274/auth/...

          resource for managing them (add auth for example). I think you'd have to require SSL to make this secure..., and return some security token good for the session so that someone else can't impersonate you... etc...

          Show
          Patrick Hunt added a comment - When I first created the REST interface I didn't have the notion of sessions, now that you do I think you would want to augment the notion of having a /znodes/... url with a url of /sessions/v1/<session TOKEN>/znodes/.... so create the session as you suggest, however that create operation returns a url representing the session, after which all of your operations use that as a "prefix" if you will. e.g.: create a new session - POST /sessions/v1?op=create HTTP/1.1 returns /sessions/v1/ab483cd8283ef274 notice the session TOKEN is a randomly generated key - this allows for some "security through obscurity" as it's "hard to guess" and is some small measure of security. session keepalive and delete would operate on this url. GET on the url might return the original session id for example create an ephemeral node - POST /sessions/v1/ab483cd8283ef274/znodes/v1/a/b?op=create&name=c&ephemeral=true HTTP/1.1 you might keep the /znodes feature as-is for those not wanting to use sessions (admin r/o console say, or cli tool), however you might want to make turning it off an option - allowing the operator to force users to use explicit sessions notice how this also cleans up items 5/6 wrt the url used to access (same prefix in both cases) when you add acl support you might add something like: /sessions/v1/ab483cd8283ef274/auth/... resource for managing them (add auth for example). I think you'd have to require SSL to make this secure..., and return some security token good for the session so that someone else can't impersonate you... etc...
          Hide
          Andrei Savu added a comment -

          This patch contains:

          • basic support for sessions ( create: POST /sessions/v1?op=create&expire=SECONDS, delete: DELETE /sessions/v1/<ID>, heartbeat: PUT /sessions/v1/<ID> )
          • support for ephemeral nodes: create: POST /znodes/v1/a/b?op=create&name=c&ephemeral=true&session=ID
          • experimental (almost complete) python REST client
          • demos: demo_master_election.py & demo_queue.py
          • updated specs

          I'm not going to implement watches now because they are not useful if the client pulls the server for changes.

          Working on:

          • configuration, ACLs and basic authentication
          • packaging - I want to be able to run the REST gateway as a task managed by the Hue [1] supervisor.

          I would really like to get more feedback from you regarding this patch. Thanks.

          [1] http://github.com/cloudera/hue

          Show
          Andrei Savu added a comment - This patch contains: basic support for sessions ( create: POST /sessions/v1?op=create&expire=SECONDS, delete: DELETE /sessions/v1/<ID>, heartbeat: PUT /sessions/v1/<ID> ) support for ephemeral nodes: create: POST /znodes/v1/a/b?op=create&name=c&ephemeral=true&session=ID experimental (almost complete) python REST client demos: demo_master_election.py & demo_queue.py updated specs I'm not going to implement watches now because they are not useful if the client pulls the server for changes. Working on: configuration, ACLs and basic authentication packaging - I want to be able to run the REST gateway as a task managed by the Hue [1] supervisor. I would really like to get more feedback from you regarding this patch. Thanks. [1] http://github.com/cloudera/hue
          Andrei Savu made changes -
          Attachment ZOOKEEPER-809.patch [ 12451218 ]
          Attachment SPEC.txt [ 12451219 ]
          Hide
          Patrick Hunt added a comment -

          Looks good to me. python and java tests are a nice touch.

          A few nits:

          spec:
          succesful -> successful
          adition -> additional
          any detail on the session uuid? format and valid characters? min/max length?
          why is session expired 503? would 400 be better?

          code:
          80char max
          try not to reformat existing code unless you're changing something in that area. makes it hard to review and confuses "blame" information

          Show
          Patrick Hunt added a comment - Looks good to me. python and java tests are a nice touch. A few nits: spec: succesful -> successful adition -> additional any detail on the session uuid? format and valid characters? min/max length? why is session expired 503? would 400 be better? code: 80char max try not to reformat existing code unless you're changing something in that area. makes it hard to review and confuses "blame" information
          Hide
          Andrei Savu added a comment -

          Thanks Patrick for reviewing the patch. I will fix those issues ASAP.

          Sorry about reformatting existing code, I've used the Eclipse Format tool and only latter I've realized that it changes the whole file.

          Show
          Andrei Savu added a comment - Thanks Patrick for reviewing the patch. I will fix those issues ASAP. Sorry about reformatting existing code, I've used the Eclipse Format tool and only latter I've realized that it changes the whole file.
          Hide
          Andrei Savu added a comment -

          Changes in this patch:

          • ZooKeeperService now uses the servlet context path and not full URI as a key for retrieving ZooKeeper connections
          • added a configuration file: conf/rest.properties
          • added support for multiple endpoints: /cluster1, /cluster2 - useful if you have multiple ZooKeeper clusters
          • HTTPS support
          • HTTP Basic Authentication
          • shutdownHook for cleanup

          I'm still working on: packaging, ACLs & ZooKeeper Authentication

          I will finish the packaging ant task tomorrow and next week (after the GSoC "soft" deadline) I will also add support for ACLs and Authentication.

          Let me know what you think. Thanks.

          Show
          Andrei Savu added a comment - Changes in this patch: ZooKeeperService now uses the servlet context path and not full URI as a key for retrieving ZooKeeper connections added a configuration file: conf/rest.properties added support for multiple endpoints: /cluster1, /cluster2 - useful if you have multiple ZooKeeper clusters HTTPS support HTTP Basic Authentication shutdownHook for cleanup I'm still working on: packaging, ACLs & ZooKeeper Authentication I will finish the packaging ant task tomorrow and next week (after the GSoC "soft" deadline) I will also add support for ACLs and Authentication. Let me know what you think. Thanks.
          Andrei Savu made changes -
          Attachment ZOOKEEPER-809.patch [ 12451335 ]
          Hide
          Andrei Savu added a comment -

          In this patch:

          • ant task for packaging: just run ant tar and it will generate zookeeper-dev-rest.tar.gz in build/contrib/rest/
          • basic script for starting and stopping the REST gateway - bin/restServer.sh

          Remaining:

          • ACLs & ZooKeeper Authentication - after GSoC "soft" deadline

          All tests are passing, including the ones from the new python client test suite.

          Show
          Andrei Savu added a comment - In this patch: ant task for packaging: just run ant tar and it will generate zookeeper-dev-rest.tar.gz in build/contrib/rest/ basic script for starting and stopping the REST gateway - bin/restServer.sh Remaining: ACLs & ZooKeeper Authentication - after GSoC "soft" deadline All tests are passing, including the ones from the new python client test suite.
          Andrei Savu made changes -
          Attachment ZOOKEEPER-809.patch [ 12451424 ]
          Hide
          Andrei Savu added a comment -

          In this patch I've added:

          • a bunch of classes for parsing the configuration file
          • per context HTTP Digest authentication settings
          • per context ZooKeeper digest authentication settings
          • support for chroot when parsing the hostPort string

          By using these features you can easily create secure channels from your application to ZooKeeper ( HTTPS + Digest Authentication + ZK Auth + chroot). It doesn't support the whole API but it should be really useful for configuration management.

          Working on:

          • support for ACLs: read and update
          • per session ZooKeeper authentication

          Sample config for a channel:

          rest.port = 9998

          rest.endpoint.1 = /channel;localhost:2181,localhost:2182,localhost:2183/app
          rest.endpoint.1.http.auth = user:pass,user2:pass2
          rest.endpoint.1.zk.digest = appuser:pass

          .. you should also enable SSL because the browser sends the password as plain text

          rest.ssl = true
          rest.ssl.jks = keys/rest.jks
          rest.ssl.jks.pass = 123456

          Show
          Andrei Savu added a comment - In this patch I've added: a bunch of classes for parsing the configuration file per context HTTP Digest authentication settings per context ZooKeeper digest authentication settings support for chroot when parsing the hostPort string By using these features you can easily create secure channels from your application to ZooKeeper ( HTTPS + Digest Authentication + ZK Auth + chroot). It doesn't support the whole API but it should be really useful for configuration management. Working on: support for ACLs: read and update per session ZooKeeper authentication Sample config for a channel: rest.port = 9998 rest.endpoint.1 = /channel;localhost:2181,localhost:2182,localhost:2183/app rest.endpoint.1.http.auth = user:pass,user2:pass2 rest.endpoint.1.zk.digest = appuser:pass .. you should also enable SSL because the browser sends the password as plain text rest.ssl = true rest.ssl.jks = keys/rest.jks rest.ssl.jks.pass = 123456
          Andrei Savu made changes -
          Attachment ZOOKEEPER-809.patch [ 12452009 ]
          Hide
          Patrick Hunt added a comment -

          This looks good, a few issues to address:

          1) add license to HTTPBasicAuth.java

          2) replace tabs with spaces for indentation (check the resulting patch file for any tabs)

          3) the diff is missing binaries for:

          :000000 100644 0000000... 13e5aab... A src/contrib/rest/conf/keys/rest.cer
          :000000 100644 0000000... 539e8be... A src/contrib/rest/conf/keys/rest.jks

          Show
          Patrick Hunt added a comment - This looks good, a few issues to address: 1) add license to HTTPBasicAuth.java 2) replace tabs with spaces for indentation (check the resulting patch file for any tabs) 3) the diff is missing binaries for: :000000 100644 0000000... 13e5aab... A src/contrib/rest/conf/keys/rest.cer :000000 100644 0000000... 539e8be... A src/contrib/rest/conf/keys/rest.jks
          Patrick Hunt made changes -
          Fix Version/s 3.4.0 [ 12314469 ]
          Hide
          Andrei Savu added a comment -

          I've added the license header to HTTPBasicAuth.java, removed all the tabs and added dummy self-signed keys to keys.tar.gz ( should be extracted to conf/keys folder ).

          Thanks Patrick for reviewing.

          Show
          Andrei Savu added a comment - I've added the license header to HTTPBasicAuth.java , removed all the tabs and added dummy self-signed keys to keys.tar.gz ( should be extracted to conf/keys folder ). Thanks Patrick for reviewing.
          Andrei Savu made changes -
          Attachment ZOOKEEPER-809.patch [ 12452253 ]
          Attachment keys.tar.gz [ 12452254 ]
          Hide
          Patrick Hunt added a comment -

          ant test in src/contrib/rest is failing:

          compile-test:
          [javac] Compiling 11 source files to /home/phunt/dev/workspace/svn_zookeeper/build/contrib/rest/test
          [javac] /home/phunt/dev/workspace/svn_zookeeper/src/contrib/rest/src/test/org/apache/zookeeper/server/jersey/Base.java:62: mapContext(java.lang.String,org.apache.zookeeper.server.jersey.cfg.Endpoint) in org.apache.zookeeper.server.jersey.ZooKeeperService cannot be applied to (java.lang.String,java.lang.String)
          [javac] ZooKeeperService.mapContext(CONTEXT_PATH, ZKHOSTPORT);
          [javac] ^
          [javac] 1 error

          Show
          Patrick Hunt added a comment - ant test in src/contrib/rest is failing: compile-test: [javac] Compiling 11 source files to /home/phunt/dev/workspace/svn_zookeeper/build/contrib/rest/test [javac] /home/phunt/dev/workspace/svn_zookeeper/src/contrib/rest/src/test/org/apache/zookeeper/server/jersey/Base.java:62: mapContext(java.lang.String,org.apache.zookeeper.server.jersey.cfg.Endpoint) in org.apache.zookeeper.server.jersey.ZooKeeperService cannot be applied to (java.lang.String,java.lang.String) [javac] ZooKeeperService.mapContext(CONTEXT_PATH, ZKHOSTPORT); [javac] ^ [javac] 1 error
          Hide
          Andrei Savu added a comment -
          • fixed text/Base.java broken by refactoring ZooKeeperService
          • all junit tests are now passing
          • all python tests are now passing
          Show
          Andrei Savu added a comment - fixed text/Base.java broken by refactoring ZooKeeperService all junit tests are now passing all python tests are now passing
          Andrei Savu made changes -
          Attachment ZOOKEEPER-809.patch [ 12452259 ]
          Hide
          Patrick Hunt added a comment -

          +1, looks great Andrei, thanks!

          Show
          Patrick Hunt added a comment - +1, looks great Andrei, thanks!
          Patrick Hunt made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in ZooKeeper-trunk #907 (See https://hudson.apache.org/hudson/job/ZooKeeper-trunk/907/)
          ZOOKEEPER-809. Improved REST Interface

          Show
          Hudson added a comment - Integrated in ZooKeeper-trunk #907 (See https://hudson.apache.org/hudson/job/ZooKeeper-trunk/907/ ) ZOOKEEPER-809 . Improved REST Interface
          Mahadev konar made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Andrei Savu
              Reporter:
              Andrei Savu
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development