[SPARK-8202] PySpark: infinite loop during external sort

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.1, 1.5.0
• Component/s: PySpark
• Target Version/s: 1.4.1, 1.5.0
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 09/Jun/15 06:30
• Updated: 18/Jun/15 20:50
• Resolved: 18/Jun/15 20:50

Description

The batch size during external sort will grow up to max 10000, then shrink down to zero, causing infinite loop.

Given the assumption that the items usually have similar size, so we don't need to adjust the batch size after first spill.

Issue Links

• links to: [Github] Pull Request #6714 (davies)