[SPARK-17877] Can not checkpoint connectedComponents resulting graph

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 1.5.2, 1.6.2, 2.0.1, 2.3.0
• Component/s: GraphX
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Alexander Pivovarov
• Created: 11/Oct/16 19:37
• Updated: 12/Dec/22 18:10
• Resolved: 08/Oct/19 05:44

Description

The following code demonstrates the issue

import org.apache.spark.graphx._
val users = sc.parallelize(List(3L -> "lucas", 7L -> "john", 5L -> "matt", 2L -> "kelly"))
val rel = sc.parallelize(List(Edge(3L, 7L, "collab"), Edge(5L, 3L, "advisor"), Edge(2L, 5L, "colleague"), Edge(5L, 7L, "pi")))
sc.setCheckpointDir("/tmp/check")

val g = Graph(users, rel)
g.checkpoint   // /tmp/check/b1f46ba5-357a-4d6d-8f4d-411b64b27c2f appears

val gg = g.connectedComponents
gg.checkpoint

gg.vertices.collect
gg.edges.collect
gg.isCheckpointed  // res5: Boolean = false,   /tmp/check still contains only 1 folder b1f46ba5-357a-4d6d-8f4d-411b64b27c2f

I think the last line should return true instead of false

Issue Links

• relates to: SPARK-14804 Graph vertexRDD/EdgeRDD checkpoint results ClassCastException:
• relates to: SPARK-12431 add local checkpointing to GraphX