Hama
  1. Hama
  2. HAMA-358

Evaluation of Hama BSP communication protocol performance

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: bsp core, documentation
    • Labels:
      None

      Description

      The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.

      There are no Sub-Tasks for this issue.

        Activity

        Hide
        Thomas Jungblut added a comment -

        Let's close this, we have benchmarks of Hadoop RPC vs. Avro here http://wiki.apache.org/hama/Benchmarks#Compare_between_Hadoop_RPC_and_Avro

        Show
        Thomas Jungblut added a comment - Let's close this, we have benchmarks of Hadoop RPC vs. Avro here http://wiki.apache.org/hama/Benchmarks#Compare_between_Hadoop_RPC_and_Avro
        Hide
        Thomas Jungblut added a comment -

        Here is avro:

        https://github.com/thomasjungblut/hama-avro/blob/master/src/de/jungblut/avro/AvroMessageServiceImpl.java

        I think the overhead might be larger than the RPC of Hadoop, but it's just a guess.

        Show
        Thomas Jungblut added a comment - Here is avro: https://github.com/thomasjungblut/hama-avro/blob/master/src/de/jungblut/avro/AvroMessageServiceImpl.java I think the overhead might be larger than the RPC of Hadoop, but it's just a guess.
        Hide
        Thomas Jungblut added a comment -

        I made a bit research with Avro and Protobuf.

        https://github.com/thomasjungblut/hama-avro

        Once HAMA-461 is done we can have a look and benchmark it.

        Show
        Thomas Jungblut added a comment - I made a bit research with Avro and Protobuf. https://github.com/thomasjungblut/hama-avro Once HAMA-461 is done we can have a look and benchmark it.
        Hide
        Thomas Jungblut added a comment -

        We can take a look at ProtoStuff:
        http://code.google.com/p/protostuff/

        I've seen that directmemory use it for serialization. Maybe it can be interesting for us too.

        Show
        Thomas Jungblut added a comment - We can take a look at ProtoStuff: http://code.google.com/p/protostuff/ I've seen that directmemory use it for serialization. Maybe it can be interesting for us too.
        Hide
        Edward J. Yoon added a comment -

        I just changed the type of this issue to research task.

        Show
        Edward J. Yoon added a comment - I just changed the type of this issue to research task.
        Hide
        Thomas Jungblut added a comment -

        I asked myself if we need to strictly stick with RPC for BSP Communication.
        What about just setting up a SequenceFile and sending it across the network using plain old streams?

        We can later add asynchronous communication during computational phases if the SequenceFile reached it's blocksize...

        What do you think of it?

        Show
        Thomas Jungblut added a comment - I asked myself if we need to strictly stick with RPC for BSP Communication. What about just setting up a SequenceFile and sending it across the network using plain old streams? We can later add asynchronous communication during computational phases if the SequenceFile reached it's blocksize... What do you think of it?
        Hide
        Edward J. Yoon added a comment -
        eddieyoon Edward J. Yoon 
        @ 
        @tjungblut By the way, serializable protobuf seems meaningless by BSPMessageBundle. What do you think?
        3 minutes ago 
        in reply to @eddieyoon ↑
        
        @tjungblut
        Thomas Jungblut
        @eddieyoon I agree, we should look into avro
        

        Copy&paste chat log.

        Show
        Edward J. Yoon added a comment - eddieyoon Edward J. Yoon @ @tjungblut By the way, serializable protobuf seems meaningless by BSPMessageBundle. What do you think? 3 minutes ago in reply to @eddieyoon ↑ @tjungblut Thomas Jungblut @eddieyoon I agree, we should look into avro Copy&paste chat log.
        Hide
        Edward J. Yoon added a comment -

        I have to look at Avro, too.

        Show
        Edward J. Yoon added a comment - I have to look at Avro, too.
        Hide
        Bill Graham added a comment -

        Could Avro help here? It doesn't require compile-time schema knowledge (you don't need to generate code stubs) so the user could define an Avro schema and then work with that at run time.

        Show
        Bill Graham added a comment - Could Avro help here? It doesn't require compile-time schema knowledge (you don't need to generate code stubs) so the user could define an Avro schema and then work with that at run time.
        Hide
        Thomas Jungblut added a comment -

        Yap, I had an older index application that runs on a server, wanted to replace RMI with this. But I gave up after two hours, it seems to be written for python primarily.
        No Was just interested in what your results might be.

        Show
        Thomas Jungblut added a comment - Yap, I had an older index application that runs on a server, wanted to replace RMI with this. But I gave up after two hours, it seems to be written for python primarily. No Was just interested in what your results might be.
        Hide
        Edward J. Yoon added a comment -

        Have you seen this project? http://code.google.com/p/protobuf-socket-rpc/

        I'm early stage. Do you want to work for this task?

        Show
        Edward J. Yoon added a comment - Have you seen this project? http://code.google.com/p/protobuf-socket-rpc/ I'm early stage. Do you want to work for this task?
        Hide
        Thomas Jungblut added a comment - - edited

        How is it going? Do you need help?

        I've practiced a bit with ProtoBuf.
        It seems to be really cool and fast (or at least small).

        BUT I have really problems with the question: how we are (or better we could be) able to integrate it.

        Let's summarize our current state:
        We have an abstract class BSPMessage which leaves the types of tags and data to the concrete implementations.
        This gets using Writable into Hadoop's RPC mechanism and gets serialized and deserialized.

        What I'm wondering now is how we can improve this using ProtoBuf.
        ProtoBuf needs a "*.proto" file that needs to be compiled to a specific model *.java file. In this file you are declaring what the message needs, in our case this is not known at compile time (for example a user implements a custom vertex that contains distances or something like that). So we have to leave the serialization up to the user.
        The question is how could we doing this?

        Here some thoughts (don't take this too serious, just some brainstorming):
        There are two options:

        • A generic ".proto" model that takes just two Strings for a tag and data
        • We leave the compiling and implementing of the protos to the user

        The first is ultra simple and we don't have to worry about anything the user will submit, since you can serialize everything to strings.
        But I think we are going to "ruin" the optimizations that COULD have been made if the type was known.

        The second option is really messy and not too user friendly (since he has to compile the whole stuff and put it into a repository at runtime that we can know the proto), but in constrast to the first option it could result in better results.

        Do we have other opportunities?

        Show
        Thomas Jungblut added a comment - - edited How is it going? Do you need help? I've practiced a bit with ProtoBuf. It seems to be really cool and fast (or at least small). BUT I have really problems with the question: how we are (or better we could be) able to integrate it. Let's summarize our current state: We have an abstract class BSPMessage which leaves the types of tags and data to the concrete implementations. This gets using Writable into Hadoop's RPC mechanism and gets serialized and deserialized. What I'm wondering now is how we can improve this using ProtoBuf. ProtoBuf needs a "*.proto" file that needs to be compiled to a specific model *.java file. In this file you are declaring what the message needs, in our case this is not known at compile time (for example a user implements a custom vertex that contains distances or something like that). So we have to leave the serialization up to the user. The question is how could we doing this? Here some thoughts (don't take this too serious, just some brainstorming): There are two options: A generic ".proto" model that takes just two Strings for a tag and data We leave the compiling and implementing of the protos to the user The first is ultra simple and we don't have to worry about anything the user will submit, since you can serialize everything to strings. But I think we are going to "ruin" the optimizations that COULD have been made if the type was known. The second option is really messy and not too user friendly (since he has to compile the whole stuff and put it into a repository at runtime that we can know the proto), but in constrast to the first option it could result in better results. Do we have other opportunities?
        Hide
        Thomas Jungblut added a comment - - edited

        I'm excited about the results ;D

        Show
        Thomas Jungblut added a comment - - edited I'm excited about the results ;D
        Hide
        Edward J. Yoon added a comment -

        I'm going to evaluate the performance of Google's protobuf.

        Show
        Edward J. Yoon added a comment - I'm going to evaluate the performance of Google's protobuf.
        Hide
        ChiaHung Lin added a comment -

        If that's possible, I would like other framework to be tested as well such as Bert (bert-rpc.org). But the main concerns is the issues related to asynchronous (or message passing) communication and the data serialization mechanism.

        Show
        ChiaHung Lin added a comment - If that's possible, I would like other framework to be tested as well such as Bert (bert-rpc.org). But the main concerns is the issues related to asynchronous (or message passing) communication and the data serialization mechanism.
        Hide
        Thomas Jungblut added a comment -

        So Zubair, you wasn't picked. Are you although going to evaluate it?

        I recently recognized that Hadoop RPC wasn't really fast. It is used at startup when the datanodes sending the blocks to the namenode. It takes 20 minutes for arround 2mio files on 6 nodes. That is really slow.

        Show
        Thomas Jungblut added a comment - So Zubair, you wasn't picked. Are you although going to evaluate it? I recently recognized that Hadoop RPC wasn't really fast. It is used at startup when the datanodes sending the blocks to the namenode. It takes 20 minutes for arround 2mio files on 6 nodes. That is really slow.
        Hide
        Zubair Nabi added a comment -

        Perfect. Thanks. I'll start writing my proposal.

        Show
        Zubair Nabi added a comment - Perfect. Thanks. I'll start writing my proposal.
        Hide
        Edward J. Yoon added a comment -

        Hadoop RPC, Thrift, and Google Protobuf are enough. and Your plan looks pretty good.

        Show
        Edward J. Yoon added a comment - Hadoop RPC, Thrift, and Google Protobuf are enough. and Your plan looks pretty good.
        Hide
        Zubair Nabi added a comment -

        Hi,

        I would like more information about this project. I have extensive background in both Hadoop, Thrift and distributed systems in general. My question is that how many frameworks do you plan on evaluating? Because the plane of RPC frameworks is pretty large. Do you have an upper bound in mind?
        Secondly, as I understand, these are going to be the steps of the project:
        1) Coming up with a list of features that are required for Hama BSP communication.
        2) Short-listing RPC frameworks based on the exhaustive list of features.
        3) Evaluation of these short-listed frameworks in a real cluster environment.
        4) Presenting this evaluation in a formal manner with graphs - in the form of a report.
        5) Recommending a final framework based on the evaluation.
        Am I missing anything?

        Thanks,
        Zubair

        Show
        Zubair Nabi added a comment - Hi, I would like more information about this project. I have extensive background in both Hadoop, Thrift and distributed systems in general. My question is that how many frameworks do you plan on evaluating? Because the plane of RPC frameworks is pretty large. Do you have an upper bound in mind? Secondly, as I understand, these are going to be the steps of the project: 1) Coming up with a list of features that are required for Hama BSP communication. 2) Short-listing RPC frameworks based on the exhaustive list of features. 3) Evaluation of these short-listed frameworks in a real cluster environment. 4) Presenting this evaluation in a formal manner with graphs - in the form of a report. 5) Recommending a final framework based on the evaluation. Am I missing anything? Thanks, Zubair
        Hide
        Edward J. Yoon added a comment -

        I'm mentoring this project.

        • name: Edward J. Yoon
        • link_id: edwardyoon
        Show
        Edward J. Yoon added a comment - I'm mentoring this project. name: Edward J. Yoon link_id: edwardyoon
        Hide
        Edward J. Yoon added a comment -
        Show
        Edward J. Yoon added a comment - Quick guide: http://wiki.apache.org/hama/GSoC2011
        Hide
        Edward J. Yoon added a comment -

        FYI, I can provide some testbed servers (200~500 nodes) If needed.

        Show
        Edward J. Yoon added a comment - FYI, I can provide some testbed servers (200~500 nodes) If needed.

          People

          • Assignee:
            Edward J. Yoon
            Reporter:
            Edward J. Yoon
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 1,008h
              1,008h
              Remaining:
              Remaining Estimate - 1,008h
              1,008h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development