Giraph
  1. Giraph
  2. GIRAPH-104

Save half of maximum memory used from messaging

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.1.0
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently, the amount of memory that Giraph uses for messaging is huge. This JIRA will reduce the messaging memory by half and provide periodic updates of memory for debugging. Details are below:

      Refactored RandomMessageBenchmark to an internal vertex class. Added aggregators to RandomMessagesBenchmark to track bytes, messages, and time for the messaging. Adjusted the postSuperstep() to be called after the flush() for more accurate timings.

      Added periodic minute updates for message flushing (which can take a while, especially on the memory benchmark). This helps to see how progress is going and gives an ETA.

      Memory optimizations include:

      • Clear the message list after computation
      • Free vertex messages on the source as the flush is going on
      • TreeMap -> HashMap for VertexMutations
      • Sizing the ArrayList properly in transientInMessages
      1. GIRAPH-104.diff
        35 kB
        Avery Ching

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Avery Ching
            Reporter:
            Avery Ching
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development