Uploaded image for project: 'Giraph'
  1. Giraph
  2. GIRAPH-874

Specialized byte array partitions

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.1.0
    • Fix Version/s: None
    • Component/s: graph
    • Labels:
      None

      Description

      While doing some performance tuning I discovered that loading byte array partitions was performing slower than expected. I found that the extra time was being spent allocating a new vertex object for each distinct vertexId encountered (because vertexId object is the map key). Similar to GIRAPH-704, the use of primitive maps can provide significant performance benefit in this situation. By using a primitive map, the vertex object on the VertexIterator can be reused perpetually because the vertexId object isn't used as the map key.

      When processing a large graph with 4B vertices the worker vertices requests were taking ~15 seconds each, but after implementing the above suggestion that number dropped down sub-second.

        Attachments

        1. GIRAPH-874-2.patch
          29 kB
          Craig Muchinsky
        2. GIRAPH-874.patch
          30 kB
          Craig Muchinsky

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              cmuchinsky@initiatesystems.com Craig Muchinsky
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: