Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1991

Support custom StorageLevels for vertices and edges

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: GraphX
    • Labels:
      None

      Description

      Large graphs may not fit entirely in memory. If we supported custom storage levels for the vertices and edges of a graph, the user could specify MEMORY_AND_DISK and then repartition the graph to use many small partitions, each of which does fit in memory. Spark would then automatically load partitions from disk as needed.

      Also, the replicated storage levels would be helpful for fault tolerance, and the serialized ones would improve efficiency for non-primitive vertex and edge attributes.

        Attachments

          Activity

            People

            • Assignee:
              ankurd Ankur Dave
              Reporter:
              ankurd Ankur Dave
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: