[SPARK-4467] Number of elements read is never reset in ExternalSorter

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.1.1, 1.2.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.1.1, 1.2.0
• Assignee: Tim
• Reporter: Andrew Or
• Created: 18/Nov/14 02:06
• Updated: 24/Nov/14 20:50
• Resolved: 19/Nov/14 18:03

Description

This is originally reported by tianshuo. This is a spin-off of the original JIRA SPARK-4452, which described a bigger problem.

Issue Links

• is duplicated by: SPARK-4515 OOM/GC errors with sort-based shuffle
• is part of: SPARK-4452 Shuffle data structures can starve others on the same thread for memory
• links to: [Github] Pull Request #3302 (tsdeng)
• links to: [Github] Pull Request #3330 (andrewor14)