Details

    • Type: Improvement
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 4.0.0
    • Component/s: None
    • Labels:
      None

      Description

      ZooKeeper currently uses Jute to serialize objects to put on the wire and on disk. We pulled Jute out of Hadoop and added a C binding. Both versions of Jute have evolved (although Hadoop still doesn't have a C binding). It would be nice to use a more standard serialization library. Some options include Thrift or Google's protocol buffers.

      Our main requirements would be Java and C bindings and good performance. (For example, serializing to XML would give us incredibly bad performance and would not be acceptible!)

        Activity

        Hide
        rgs Raul Gutierrez Segales added a comment -

        We don't necessarily have to break other clients. Can't we not use the protocol field in the ConnectRequest to negotiate such things as serialization (or compression, etc... it's a 32bit after all, lots of space!)?

        Show
        rgs Raul Gutierrez Segales added a comment - We don't necessarily have to break other clients. Can't we not use the protocol field in the ConnectRequest to negotiate such things as serialization (or compression, etc... it's a 32bit after all, lots of space!)?
        Hide
        michim Michi Mutsuzaki added a comment -

        Reopened the issue for 4.0.0. I am worried about breaking compatibility through, given that it'll break third party client libraries like kazoo.

        Show
        michim Michi Mutsuzaki added a comment - Reopened the issue for 4.0.0. I am worried about breaking compatibility through, given that it'll break third party client libraries like kazoo.
        Hide
        fpj Flavio Junqueira added a comment -

        I think we should leave this open. It does seem like a tough one at this point, but I still have hope that it can happen some day. We should consider doing a 4.0.0 version so that we can break compatibility.

        Show
        fpj Flavio Junqueira added a comment - I think we should leave this open. It does seem like a tough one at this point, but I still have hope that it can happen some day. We should consider doing a 4.0.0 version so that we can break compatibility.
        Hide
        michim Michi Mutsuzaki added a comment -

        I'm closing this as wontfix for now. It would be great to be able to use more modern serialization libraries, but it doesn't seem realistic to do it in a backward compatible way.

        Show
        michim Michi Mutsuzaki added a comment - I'm closing this as wontfix for now. It would be great to be able to use more modern serialization libraries, but it doesn't seem realistic to do it in a backward compatible way.
        Hide
        fournc Camille Fournier added a comment -

        Do we really care enough to make this worth doing? It's been 6 years...

        Show
        fournc Camille Fournier added a comment - Do we really care enough to make this worth doing? It's been 6 years...
        Hide
        abranzyck Germán Blanco added a comment -

        Ooops, should have read the entire thread ... so maybe there could be ways to enable the upgrade after all.

        Show
        abranzyck Germán Blanco added a comment - Ooops, should have read the entire thread ... so maybe there could be ways to enable the upgrade after all.
        Hide
        abranzyck Germán Blanco added a comment -

        Avro seems to have a better support for modifying the message and adding optional elements without breaking backwards compatibility. I think that would be nice to have.
        But of course, changing to Avro (or anything else) would mean breaking the upgrade between two releases. So it is not an easy decision.

        Show
        abranzyck Germán Blanco added a comment - Avro seems to have a better support for modifying the message and adding optional elements without breaking backwards compatibility. I think that would be nice to have. But of course, changing to Avro (or anything else) would mean breaking the upgrade between two releases. So it is not an easy decision.
        Hide
        nileader Leader Ni added a comment -

        So, how about avro now in hadoop,and any paln to replace jute?

        Show
        nileader Leader Ni added a comment - So, how about avro now in hadoop,and any paln to replace jute?
        Hide
        phunt Patrick Hunt added a comment -

        haha, doug and I crossed beams on comments.

        Show
        phunt Patrick Hunt added a comment - haha, doug and I crossed beams on comments.
        Hide
        phunt Patrick Hunt added a comment -

        avro is wire compatible with jute?

        in general no, however Doug was wondering if there was some way to define the avro spec in such a way that it would work.

        Would this be a good Google Summer of Code task?

        it would be an interesting project regardless. We've discussed it a few times, the general idea being that we should add the ability to configure a new client port using Avro, maintain b/w compatibility by leaving the old jute based port alone (although allow it to be turned off through config). Once all clients were migrated the zk ensemble admin could turn off the legacy jute port.

        Btw, while fixing this we should also split the client and admin (4letterword) ports. This causes us all kinds of headache (today shared) and really the 4letterword feature should be expanded to allow for more sophisticated usage (for example long lived connection). Also I've heard complaints from admins that they don't want to expose the 4letter words to all users (it's r/o but they want to lock down that data). Again, having a separate admin more might allow for more features.

        One other issue comes to mind. Last time I benchmarked our message definitions on Jute vs Avro the avro C client performance was terrible. We wouldn't want to switch over from jute to avro (default I'm saying) until this is addressed. I'm also not sure if the avro c api is locked down.

        Show
        phunt Patrick Hunt added a comment - avro is wire compatible with jute? in general no, however Doug was wondering if there was some way to define the avro spec in such a way that it would work. Would this be a good Google Summer of Code task? it would be an interesting project regardless. We've discussed it a few times, the general idea being that we should add the ability to configure a new client port using Avro, maintain b/w compatibility by leaving the old jute based port alone (although allow it to be turned off through config). Once all clients were migrated the zk ensemble admin could turn off the legacy jute port. Btw, while fixing this we should also split the client and admin (4letterword) ports. This causes us all kinds of headache (today shared) and really the 4letterword feature should be expanded to allow for more sophisticated usage (for example long lived connection). Also I've heard complaints from admins that they don't want to expose the 4letter words to all users (it's r/o but they want to lock down that data). Again, having a separate admin more might allow for more features. One other issue comes to mind. Last time I benchmarked our message definitions on Jute vs Avro the avro C client performance was terrible. We wouldn't want to switch over from jute to avro (default I'm saying) until this is addressed. I'm also not sure if the avro c api is locked down.
        Hide
        cutting Doug Cutting added a comment -

        Avro is not wire-compatible with Jute.

        Show
        cutting Doug Cutting added a comment - Avro is not wire-compatible with Jute.
        Hide
        thkoch Thomas Koch added a comment -

        Another option would be avro. Does anybody know, whether avro is wire compatible with jute? We would need to support communication between older jute-only Clients/Servers and newer avro(+jute?) Clients/Servers.

        Would this be a good Google Summer of Code task?

        Show
        thkoch Thomas Koch added a comment - Another option would be avro. Does anybody know, whether avro is wire compatible with jute? We would need to support communication between older jute-only Clients/Servers and newer avro(+jute?) Clients/Servers. Would this be a good Google Summer of Code task?
        Hide
        chirino Hiram Chirino added a comment -

        protobuf might be an option.

        Show
        chirino Hiram Chirino added a comment - protobuf might be an option.

          People

          • Assignee:
            Unassigned
            Reporter:
            breed Benjamin Reed
          • Votes:
            1 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:

              Development