Uploaded image for project: 'Giraph (Retired)'
  1. Giraph (Retired)
  2. GIRAPH-322

Run Length Encoding for Vertex#sendMessageToAllEdges might curb out of control message growth in large scale jobs

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Won't Fix
    • 1.0.0
    • 1.0.0
    • bsp
    • None

    Description

      Vertex#sendMessageToAllEdges is a case that goes against the grain of the data structures and code paths used to transport messages through a Giraph application and out on the network. While messages to a single vertex can be combined (and should be) in some applications that could make use of this broadcast messaging, the out of control message growth of algorithms like triangle closing means we need to de-duplicate messages bound for many vertices/partitions.

      This will be an evolving solution (this first patch is just the first step) and currently it does not present a robust solution for disk-spill message stores. I figure I can get some advice about that or it can be a follow-up JIRA if this turns out to be a fruitful pursuit. This first patch is also Netty-only and simply defaults to the old sendMessagesToAllEdges() implementation if USE_NETTY is false. All this can be cleaned up when we know this works and/or is worth pursuing.

      The idea is to send as few broadcast messages as possible by run-length encoding their delivery and only duplicating message on the network when they are bound for different partitions. This is also best when combined with "-Dhash.userPartitionCount=# of workers" so you don't do too much of that.

      If this shows promise I will report back and keep working on this. As it is, it represents an end-to-end solution, using Netty, for in-memory messaging. It won't break with spill to disk, but you do lose the de-duplicating effect.

      More to follow, comments/ideas welcome. I expect this to change a lot as I test it and ideas/suggestions crop up.

      Attachments

        1. GIRAPH-322-1.patch
          28 kB
          Eli Reisman
        2. GIRAPH-322-2.patch
          28 kB
          Eli Reisman
        3. GIRAPH-322-3.patch
          28 kB
          Eli Reisman
        4. GIRAPH-322-4.patch
          27 kB
          Eli Reisman
        5. GIRAPH-322-5.patch
          28 kB
          Eli Reisman
        6. GIRAPH-322-6.patch
          28 kB
          Eli Reisman

        Issue Links

          Activity

            People

              initialcontext Eli Reisman
              initialcontext Eli Reisman
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: