[SPARK-10546] Check partitionId's range in ExternalSorter#spill()

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 1.4.1
• Fix Version/s: 1.6.0
• Component/s: Spark Core
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 10/Sep/15 19:17
• Updated: 11/Sep/15 20:46
• Resolved: 11/Sep/15 20:46

Description

See this thread for background:
http://search-hadoop.com/m/q3RTt0rWvIkHAE81

We should check the range of partition Id and provide meaningful message through exception.

Alternatively, we can use abs() and modulo to force the partition Id into legitimate range. However, expectation is that user should correct the logic error in his / her code.

Issue Links

• links to: [Github] Pull Request #8703 (tedyu)