[SPARK-451] Ranges of Longs don't get split properly by SparkContext.parallelize

• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Labels: Starter
• Assignee: Unassigned
• Reporter: Matei Alexandru Zaharia
• Created: 31/Oct/11 09:49
• Updated: 10/Oct/13 18:00
• Resolved: 19/Oct/12 22:50

Description

We wanted Ranges to be split into smaller Range objects to avoid shipping the integers in them over the network, but Scala 2.9's Long ranges don't get matched as a Range by the ParallelCollection code and thus get converted to arrays, costing a lot of RAM.