Uploaded image for project: 'Giraph (Retired)'
  1. Giraph (Retired)
  2. GIRAPH-249

Move part of the graph out-of-core when memory is low

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • None
    • None

    Description

      There has been some talk about Giraph's scaling limitations due to keeping the whole graph and messages in RAM.
      We need to investigate methods to fall back to disk when running out of memory, while gracefully degrading performance.

      This issue is for graph storage. Messages should probably be a separate issue, although the interplay between the two is crucial.

      We should also discuss what are our primary goals here: completing a job (albeit slowly) instead of failing when the graph is too big, while still encouraging memory optimizations and high-memory clusters; or restructuring Giraph to be as efficient as possible in disk mode, making it almost a standard way of operating.

      Attachments

        1. GIRAPH-249.patch
          73 kB
          Alessandro Presta
        2. GIRAPH-249.patch
          74 kB
          Alessandro Presta
        3. GIRAPH-249.patch
          70 kB
          Alessandro Presta
        4. GIRAPH-249.patch
          70 kB
          Alessandro Presta
        5. GIRAPH-249.patch
          70 kB
          Alessandro Presta
        6. GIRAPH-249.patch
          68 kB
          Alessandro Presta
        7. GIRAPH-249.patch
          67 kB
          Alessandro Presta
        8. GIRAPH-249.patch
          22 kB
          Alessandro Presta
        9. GIRAPH-249.patch
          22 kB
          Alessandro Presta
        10. GIRAPH-249.patch
          22 kB
          Alessandro Presta
        11. GIRAPH-249.patch
          22 kB
          Alessandro Presta
        12. GIRAPH-249.patch
          21 kB
          Alessandro Presta

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. GIRAPH-45 Improve the way to keep outgoing messages

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. GIRAPH-247 Introduce edge based partitioning for InputSplits

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              apresta Alessandro Presta
              apresta Alessandro Presta
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: