[SPARK-23358] When the number of partitions is greater than 2^28, it will result in an error result

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.2.2, 2.3.0
• Component/s: Spark Core
• Assignee: liuxian
• Reporter: liuxian
• Created: 08/Feb/18 11:29
• Updated: 12/Feb/18 14:33
• Resolved: 09/Feb/18 14:45

Description

In the `checkIndexAndDataFile`,the blocks is the  Int type,  when it is greater than 2^28, `blocks*8` will overflow, and this will result in an error result.
In fact, `blocks` is actually the number of partitions.

Issue Links

• is related to: SPARK-23391 It may lead to overflow for some integer multiplication
• links to: [Github] Pull Request #20544 (10110346)