Uploaded image for project: 'Giraph'
  1. Giraph
  2. GIRAPH-873

Specialized edge stores

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.0
    • Component/s: None
    • Labels:
      None

      Description

      While doing some performance tuning I discovered that loading the edge store can be a very expensive operation. Similar to GIRAPH-704, the use of primitive maps can provide significant performance benefit. Part of the benefit comes with the lower memory overhead associated with the primitive maps however the larger benefit comes with the fact that you don't have to release and reconstruct the vertexId object every time a new vertex is encountered.

      When processing a large graph with 4B vertices and 5B edges (3B of the edges loaded via EdgeInputFormat) the worker edge requests were taking ~15 seconds each, but after implementing the above suggestions that number dropped down sub-second.

        Attachments

        1. GIRAPH-873.patch
          41 kB
          Craig Muchinsky
        2. GIRAPH-873-2.patch
          41 kB
          Craig Muchinsky
        3. GIRAPH-873_refactor.patch
          45 kB
          Pavan Kumar

          Activity

            People

            • Assignee:
              pavanka Pavan Kumar
              Reporter:
              cmuchinsky@initiatesystems.com Craig Muchinsky
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: