[SPARK-3190] Creation of large graph(> 2.15 B nodes) seems to be broken:possible overflow somewhere

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.3
• Fix Version/s: 1.0.3, 1.1.0, 1.2.0, 1.3.2, 1.4.2, 1.5.0
• Component/s: GraphX

Environment

Standalone mode running on EC2 . Using latest code from master branch upto commit #db56f2df1b8027171da1b8d2571d1f2ef1e103b6 .

• Target Version/s: 1.3.2, 1.4.2, 1.5.0
• Assignee: Ankur Dave
• Reporter: npanj
• Created: 23/Aug/14 00:29
• Updated: 02/May/16 06:20
• Resolved: 04/Aug/15 06:08

Description

While creating a graph with 6B nodes and 12B edges, I noticed that 'numVertices' api returns incorrect result; 'numEdges' reports correct number. For few times(with different dataset > 2.5B nodes) I have also notices that numVertices is returned as -ive number; so I suspect that there is some overflow (may be we are using Int for some field?).

Here is some details of experiments I have done so far:
1. Input: numNodes=6101995593 ; noEdges=12163784626
Graph returns: numVertices=1807028297 ; numEdges=12163784626

2. Input : numNodes=2157586441 ; noEdges=2747322705
Graph Returns: numVertices=-2137380855 ; numEdges=2747322705

3. Input: numNodes=1725060105 ; noEdges=204176821
Graph: numVertices=1725060105 ; numEdges=2041768213

You can find the code to generate this bug here:

https://gist.github.com/npanj/92e949d86d08715bf4bf

Note: Nodes are labeled are 1...6B .

Issue Links

• is duplicated by: SPARK-10228 Integer overflow in VertexRDDImpl.count
• links to: [Github] Pull Request #2106 (ankurdave)
• links to: [Github] Pull Request #7923 (ankurdave)
• links to: [Github] Pull Request #12835 (liyuance)