Uploaded image for project: 'Giraph'
  1. Giraph
  2. GIRAPH-1025

Flow-control mechanism improving out-of-core feature

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: graph
    • Labels:
      None

      Description

      Current implementation of out-of-core mechanism may fail due to message flooding in applications without message combiner. In such applications the rate at which messages arrive may be much more than the rate of read/write to disk in OOC mechanism. In this scenario, the OOC mechanism will get OOM as it cannot keep up with storing incoming messages in-memory.

      One might say this problem may also appear in input-superstep as we have input vertices and edges going around as messages, but note that in input superstep the rate at which a worker receive messages (vertices and edges) are almost the same as the rate of reading from disk, and that is almost the same as the rate at which offloading data to disk happens. However, this flow-control mechanism can be generalized for input superstep too. At this point, this generalization does not seem necessary though.

        Attachments

          Activity

            People

            • Assignee:
              heslami Hassan Eslami
              Reporter:
              heslami Hassan Eslami
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: