[SPARK-10228] Integer overflow in VertexRDDImpl.count

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.4.1
• Component/s: GraphX
• Assignee: Unassigned
• Reporter: Robin Cheng
• Created: 25/Aug/15 16:37
• Updated: 25/Aug/15 16:57
• Resolved: 25/Aug/15 16:57

Description

VertexRDDImpl overrides RDD.count() but aggregates Int instead of Long:

/** The number of vertices in the RDD. */
override def count(): Long =

{ partitionsRDD.map(_.size).reduce(_ + _) }

This causes Pregel to stop iterating when the number of messages is "negative", giving incorrect results.

Issue Links

• duplicates: SPARK-3190 Creation of large graph(> 2.15 B nodes) seems to be broken:possible overflow somewhere