Spark
  1. Spark
  2. SPARK-2173

Add Master Computer and SuperStep Accumulator to Pregel GraphX Implemention

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      In Girpah there is an idea of a master compute and a global superstep value you can access. I would like to add that to GraphX.

      Let me know what you think. I will try to get a pull request tonight.

        Activity

        Hide
        Ted Malaska added a comment -

        This is my first attempt. Let me know what you think.

        https://github.com/apache/spark/pull/1111

        Thanks

        Show
        Ted Malaska added a comment - This is my first attempt. Let me know what you think. https://github.com/apache/spark/pull/1111 Thanks
        Hide
        Ted Malaska added a comment -

        Never mind. I had to close the pull request. I thought about it. The ccAccumulator is not accessible from the vprog which was my goal. I'm going to have to use a broadcast. I will have an update for tomorrow.

        Show
        Ted Malaska added a comment - Never mind. I had to close the pull request. I thought about it. The ccAccumulator is not accessible from the vprog which was my goal. I'm going to have to use a broadcast. I will have an update for tomorrow.
        Hide
        Ted Malaska added a comment -

        Nope a broadcast wont work either. Let me think about it over night. Maybe the solution is simply just update the VertexRDD.innerJoin method to take i which is the super step.

        Show
        Ted Malaska added a comment - Nope a broadcast wont work either. Let me think about it over night. Maybe the solution is simply just update the VertexRDD.innerJoin method to take i which is the super step.
        Hide
        Ted Malaska added a comment -

        Sorry I'm slow I just realized I don't even need superStep of master computer. In the confines of Giraph I did need them to solve tree rooting, but in the world of GraphX and can enter and exit Pregel when ever and how often I want.

        So in the case of tree rooting. I would do max one super step of pregel to broadcast to all my children to identify my roots then start a second pregel with un bound super steps to root all the other vertices to the roots.

        GraphX is so freeing in comparison to Giraph.

        I will close the ticket now.

        Show
        Ted Malaska added a comment - Sorry I'm slow I just realized I don't even need superStep of master computer. In the confines of Giraph I did need them to solve tree rooting, but in the world of GraphX and can enter and exit Pregel when ever and how often I want. So in the case of tree rooting. I would do max one super step of pregel to broadcast to all my children to identify my roots then start a second pregel with un bound super steps to root all the other vertices to the roots. GraphX is so freeing in comparison to Giraph. I will close the ticket now.
        Hide
        Ted Malaska added a comment -

        Not an issue. GraphX doesn't need these features because it is not as limiting at Giraph in it's options.

        Show
        Ted Malaska added a comment - Not an issue. GraphX doesn't need these features because it is not as limiting at Giraph in it's options.
        Hide
        Matthew Cornell added a comment -

        Hello. I'm new to GraphX and Scala and my sendMsg function needs to index into an input list to my algorithm based on the pregel()() iteration number, but I don't see a way to access that. It sounds like you worked around it somehow, but I'm not sure what you mean. Are you running the pregel() method separately one step at a time? Thank you.

        Show
        Matthew Cornell added a comment - Hello. I'm new to GraphX and Scala and my sendMsg function needs to index into an input list to my algorithm based on the pregel()() iteration number, but I don't see a way to access that. It sounds like you worked around it somehow, but I'm not sure what you mean. Are you running the pregel() method separately one step at a time? Thank you.

          People

          • Assignee:
            Unassigned
            Reporter:
            Ted Malaska
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development