Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3623

Graph should support the checkpoint operation

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.2, 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: GraphX
    • Labels:
      None

      Description

      Consider the following code:

          for (i <- 0 until totalIter) {
            val previousCorpus = corpus
            logInfo("Start Gibbs sampling (Iteration %d/%d)".format(i, totalIter))
      
            val corpusTopicDist = collectTermTopicDist(corpus, globalTopicCounter, sumTerms,
              numTerms, numTopics, alpha, beta).persist(storageLevel)
      
            val corpusSampleTopics = sampleTopics(corpusTopicDist, globalTopicCounter, sumTerms, numTerms,
              numTopics, alpha, beta).persist(storageLevel)
      
            corpus = updateCounter(corpusSampleTopics, numTopics).persist(storageLevel)
      
            globalTopicCounter = collectGlobalCounter(corpus, numTopics)
            assert(bsum(globalTopicCounter) == sumTerms)
            previousCorpus.unpersistVertices()
            corpusTopicDist.unpersistVertices()
            corpusSampleTopics.unpersistVertices()
          }
      

      If there is no checkpoint operation will appear the following problems.
      1. The RDD of corpus dependencies are too deep
      2. The shuffle files are too large.
      3. Any of a server crash will cause the algorithm to recalculate

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gq Guoqiang Li
                Reporter:
                gq Guoqiang Li
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: