[SPARK-6697] PeriodicGraphCheckpointer is not clear edges.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.3.0
• Component/s: GraphX, MLlib
• Assignee: Unassigned
• Reporter: Guoqiang Li
• Created: 03/Apr/15 15:29
• Updated: 10/Sep/15 01:30
• Resolved: 10/Sep/15 01:30

Description

When I run this branch(lrGraphxSGD) .
PeriodicGraphCheckpointer only clear the vertices.

 
def run(iterations: Int): Unit = {
    for (iter <- 1 to iterations) {
      logInfo(s"Start train (Iteration $iter/$iterations)")
      val margin = forward()
      margin.setName(s"margin-$iter").persist(storageLevel)
      println(s"train (Iteration $iter/$iterations) cost : ${error(margin)}")
      var gradient = backward(margin)
      gradient = updateDeltaSum(gradient, iter)

      dataSet = updateWeight(gradient, iter)
      dataSet.vertices.setName(s"vertices-$iter")
      dataSet.edges.setName(s"edges-$iter")
      dataSet.persist(storageLevel)
      graphCheckpointer.updateGraph(dataSet)

      margin.unpersist(blocking = false)
      gradient.unpersist(blocking = false)
      logInfo(s"End train (Iteration $iter/$iterations)")
      innerIter += 1
    }
    graphCheckpointer.deleteAllCheckpoints()
  }

  // Updater for L1 regularized problems
  private def updateWeight(delta: VertexRDD[Double], iter: Int): Graph[VD, ED] = {
    val thisIterStepSize = if (useAdaGrad) stepSize else stepSize / sqrt(iter)
    val thisIterL1StepSize = stepSize / sqrt(iter)
    val newVertices = dataSet.vertices.leftJoin(delta) { (_, attr, gradient) =>
      gradient match {
        case Some(gard) => {
          var weight = attr
          weight -= thisIterStepSize * gard
          if (regParam > 0.0 && weight != 0.0) {
            val shrinkageVal = regParam * thisIterL1StepSize
            weight = signum(weight) * max(0.0, abs(weight) - shrinkageVal)
          }
          assert(!weight.isNaN)
          weight
        }
        case None => attr
      }
    }
    GraphImpl(newVertices, dataSet.edges)
  }

Issue Links

• relates to: SPARK-3625 In some cases, the RDD.checkpoint does not work