Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5561

Generalize PeriodicGraphCheckpointer for RDDs

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.5.0
    • Component/s: GraphX, MLlib, Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      PeriodicGraphCheckpointer was introduced for Latent Dirichlet Allocation (LDA), but it could be generalized to work with both Graphs and RDDs. It should be generalized and moved out of MLlib.

      (For those who are not familiar with it, it tries to automatically handle persisting/unpersisting and checkpointing/removing checkpoint files in a lineage of Graphs.)

      A generalized version might be immediately useful for:

      • RandomForest
      • Streaming
      • GLMs

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                josephkb Joseph K. Bradley
                Reporter:
                josephkb Joseph K. Bradley
                Shepherd:
                Xiangrui Meng
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: