Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-28

Make Apache Tajo works with Protocol buffer 2.5.0

    Details

    • Type: Wish
    • Status: Resolved
    • Priority: Minor
    • Resolution: Later
    • Affects Version/s: None
    • Fix Version/s: 0.2-incubating
    • Component/s: RPC
    • Labels:
      None

      Description

      Early test indicates the code fail to compile with protocol buffer 2.5.0.

      Looks like we need to make it work with latest proto buffer version (2.5.0) because of bug in gtest that prevent it to be install in some linux versions:
      https://code.google.com/p/protobuf/issues/detail?id=433

        Activity

        Hide
        hsaputra Henry Saputra added a comment -

        Better yet, maybe we could replace it with Apache Avro to avoid code generation if possible

        Show
        hsaputra Henry Saputra added a comment - Better yet, maybe we could replace it with Apache Avro to avoid code generation if possible
        Hide
        initialcontext Eli Reisman added a comment -

        Switching to Avro would be a bit more work, but this is a new project and now is the time to do stuff like this. Crunch and others that use Avro internally have been very very happy with it. I am very +1 on that approach.

        I also think if we don't have time right now, bumping our protobuf version is a fine temporary solution to avoid crashing people's installs!

        Anyone else have an opinion on this?

        Show
        initialcontext Eli Reisman added a comment - Switching to Avro would be a bit more work, but this is a new project and now is the time to do stuff like this. Crunch and others that use Avro internally have been very very happy with it. I am very +1 on that approach. I also think if we don't have time right now, bumping our protobuf version is a fine temporary solution to avoid crashing people's installs! Anyone else have an opinion on this?
        Hide
        initialcontext Eli Reisman added a comment -

        You know, reading that Google thread link, it seems the problem is with gtest, and they attempted to patch by including a guest 1.5 version with the latest 2.4.1 protobuf. We should see if this fixes it. Most of the other tools in the community run fine on linux boxes on 2.4.1 which is why I was surprised. Maybe we need to run guest on our current dep and see what happens?

        On the other hand, I'm very interested in having the Avro discussion, so its good we brought this up anyway.

        Show
        initialcontext Eli Reisman added a comment - You know, reading that Google thread link, it seems the problem is with gtest, and they attempted to patch by including a guest 1.5 version with the latest 2.4.1 protobuf. We should see if this fixes it. Most of the other tools in the community run fine on linux boxes on 2.4.1 which is why I was surprised. Maybe we need to run guest on our current dep and see what happens? On the other hand, I'm very interested in having the Avro discussion, so its good we brought this up anyway.
        Hide
        hsaputra Henry Saputra added a comment - - edited

        Yeah, we should definitely bring the Apache Avro discussion in another JIRA/thread =)

        As for protobuf issue, I did pass the issue patching the gtest with 1.5 and mock the code in the protobuf a but to avoid exception.

        There is only one version of 2.4.1 in maven central: http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22com.google.protobuf%22%20AND%20a%3A%22protobuf-java%22
        and last update is 06-Jul-2011 so probably wont have the gtest update.
        I think they meant they update the gtest in the trunk which is the latest 2.5.0 update.

        I think moving to 2.5.0 is better approach than dwelling with older artifact.

        Show
        hsaputra Henry Saputra added a comment - - edited Yeah, we should definitely bring the Apache Avro discussion in another JIRA/thread =) As for protobuf issue, I did pass the issue patching the gtest with 1.5 and mock the code in the protobuf a but to avoid exception. There is only one version of 2.4.1 in maven central: http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22com.google.protobuf%22%20AND%20a%3A%22protobuf-java%22 and last update is 06-Jul-2011 so probably wont have the gtest update. I think they meant they update the gtest in the trunk which is the latest 2.5.0 update. I think moving to 2.5.0 is better approach than dwelling with older artifact.
        Hide
        initialcontext Eli Reisman added a comment -

        Got it. that makes sense. I wonder if we could ping them to update the
        Maven repo now that Google itself has fixed the problem...

        Show
        initialcontext Eli Reisman added a comment - Got it. that makes sense. I wonder if we could ping them to update the Maven repo now that Google itself has fixed the problem...
        Hide
        hyunsik Hyunsik Choi added a comment -

        +1 for switching to Avro.

        Actually, I have considered the switch from protocol buffer to Avro. In addition, tajo-rpc could be replaced by avro-rpc. If we switch tajo-rpc, we can concentrate on fewer works. It may be the right time to do them because Tajo is under early stage and is an incubator project.

        It will be very tedious work because many (de)serializable objects are implemented by protocol buffers. It would be good to create many subtasks for them. If there are not any objection until this weekend, I'll an umbrella jira issue for subtasks. It also would be a very good opportunity to create issues for newbies.

        Show
        hyunsik Hyunsik Choi added a comment - +1 for switching to Avro. Actually, I have considered the switch from protocol buffer to Avro. In addition, tajo-rpc could be replaced by avro-rpc. If we switch tajo-rpc, we can concentrate on fewer works. It may be the right time to do them because Tajo is under early stage and is an incubator project. It will be very tedious work because many (de)serializable objects are implemented by protocol buffers. It would be good to create many subtasks for them. If there are not any objection until this weekend, I'll an umbrella jira issue for subtasks. It also would be a very good opportunity to create issues for newbies.
        Hide
        hsaputra Henry Saputra added a comment -

        +1

        Thanks Hyunsik, and if it is not already obvious I would like to help with the Avro migration stories =)

        Show
        hsaputra Henry Saputra added a comment - +1 Thanks Hyunsik, and if it is not already obvious I would like to help with the Avro migration stories =)
        Hide
        hyunsik Hyunsik Choi added a comment -

        Thank you for Henry. Yes, you can start Avro migration

        Show
        hyunsik Hyunsik Choi added a comment - Thank you for Henry. Yes, you can start Avro migration
        Hide
        hsaputra Henry Saputra added a comment -

        Hyunsik, since you have deep knowledge about Tajo, the subtasks and the umbrella jira issue will be super helpful to divide and guide the transformation.
        We may need a branch to do the coding I suppose.

        Show
        hsaputra Henry Saputra added a comment - Hyunsik, since you have deep knowledge about Tajo, the subtasks and the umbrella jira issue will be super helpful to divide and guide the transformation. We may need a branch to do the coding I suppose.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Henry Saputra Ok, I'll create an umbrella jira issue and subtasks in this weekend. Thank you!

        Show
        hyunsik Hyunsik Choi added a comment - Henry Saputra Ok, I'll create an umbrella jira issue and subtasks in this weekend. Thank you!
        Hide
        hyunsik Hyunsik Choi added a comment -

        Aside from the switch issue of protocol buffer, PB's bug looks temporarily. If protocol buffer 2.5.0 has some bug, we can postpone the upgrade of protocol buffer. Do you have some problem with gtest?

        Show
        hyunsik Hyunsik Choi added a comment - Aside from the switch issue of protocol buffer, PB's bug looks temporarily. If protocol buffer 2.5.0 has some bug, we can postpone the upgrade of protocol buffer. Do you have some problem with gtest?
        Hide
        hsaputra Henry Saputra added a comment -

        There is an issue with PB 2.4.1 getest that requires update. However the fix only comes with PB 2.5.0 and they don't create new release for 2.4.x.

        This bug is filed to move Tajo to work with PB 2.5.0 before we are able to switch to Avro

        Show
        hsaputra Henry Saputra added a comment - There is an issue with PB 2.4.1 getest that requires update. However the fix only comes with PB 2.5.0 and they don't create new release for 2.4.x. This bug is filed to move Tajo to work with PB 2.5.0 before we are able to switch to Avro
        Hide
        hyunsik Hyunsik Choi added a comment -

        I don't have any background about gtest. I'm sorry for that. Could you explain how gtest is related to Tajo. Do you mean 2.4.1 have some critical problem? If not, in my opinion, we should postpone to bump up protocol buffer to 2.5.0 because recent stable linux distributions only provides protocol buffer 2.4.1.

        Show
        hyunsik Hyunsik Choi added a comment - I don't have any background about gtest. I'm sorry for that. Could you explain how gtest is related to Tajo. Do you mean 2.4.1 have some critical problem? If not, in my opinion, we should postpone to bump up protocol buffer to 2.5.0 because recent stable linux distributions only provides protocol buffer 2.4.1.
        Hide
        hsaputra Henry Saputra added a comment -

        You are right, I will downgrade the issue and punt it for now. We could live with 2.4.1 PB.

        The issue happened when you tried to make C++ component for the PB in Linux due to gtest fail. Its issue with gtest package with PB 2.4.1 not really Tajo issue.

        I think we should invest more on moving to Apache Avro.

        Show
        hsaputra Henry Saputra added a comment - You are right, I will downgrade the issue and punt it for now. We could live with 2.4.1 PB. The issue happened when you tried to make C++ component for the PB in Linux due to gtest fail. Its issue with gtest package with PB 2.4.1 not really Tajo issue. I think we should invest more on moving to Apache Avro.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Thank you for your understanding.

        Show
        hyunsik Hyunsik Choi added a comment - Thank you for your understanding.

          People

          • Assignee:
            hsaputra Henry Saputra
            Reporter:
            hsaputra Henry Saputra
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development