[SPARK-25234] SparkR:::parallelize doesn't handle integer overflow properly

• Type: Story
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.1, 2.4.0
• Fix Version/s: 2.3.2, 2.4.0
• Component/s: SparkR
• Assignee: Xiangrui Meng
• Reporter: Xiangrui Meng
• Created: 24/Aug/18 16:46
• Updated: 24/Aug/18 22:04
• Resolved: 24/Aug/18 22:04

Description

parallelize uses integer multiplication, which cannot handle size over ~47000. This cause issues with lapply

 

SparkR:::parallelize(sc, 1:47000, 47000)

Error in rep(start, end - start) : invalid 'times' argument
Error in rep(start, end - start) : invalid 'times' argument
In addition: Warning message:
In x * length(coll) : NAs produced by integer overflow

Issue Links

• links to: [Github] Pull Request #22225 (mengxr)