Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1991

Support custom StorageLevels for vertices and edges

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.1.0
    • GraphX
    • None

    Description

      Large graphs may not fit entirely in memory. If we supported custom storage levels for the vertices and edges of a graph, the user could specify MEMORY_AND_DISK and then repartition the graph to use many small partitions, each of which does fit in memory. Spark would then automatically load partitions from disk as needed.

      Also, the replicated storage levels would be helpful for fault tolerance, and the serialized ones would improve efficiency for non-primitive vertex and edge attributes.

      Attachments

        Activity

          People

            ankurd Ankur Dave
            ankurd Ankur Dave
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: