[SPARK-1931] Graph.partitionBy does not reconstruct routing tables

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.1
• Component/s: GraphX
• Assignee: Ankur Dave
• Reporter: Ankur Dave
• Created: 26/May/14 17:45
• Updated: 29/May/14 00:11
• Resolved: 27/May/14 01:31

Description

Commit 905173df57b90f90ebafb22e43f55164445330e6 introduced a bug in partitionBy where, after repartitioning the edges, it reuses the VertexRDD without updating the routing tables to reflect the new edge layout. This causes the following test to fail:

      import org.apache.spark.graphx._
      val g = Graph(
        sc.parallelize(List((0L, "a"), (1L, "b"), (2L, "c"))),
        sc.parallelize(List(Edge(0L, 1L, 1), Edge(0L, 2L, 1)), 2))
      assert(g.triplets.collect.map(_.toTuple).toSet ==
        Set(((0L, "a"), (1L, "b"), 1), ((0L, "a"), (2L, "c"), 1)))
      val gPart = g.partitionBy(PartitionStrategy.EdgePartition2D)
      assert(gPart.triplets.collect.map(_.toTuple).toSet ==
        Set(((0L, "a"), (1L, "b"), 1), ((0L, "a"), (2L, "c"), 1)))