[SPARK-21243] Limit the number of maps in a single shuffle fetch

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.1.1
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: Spark Core
• Assignee: Dhruve Ashar
• Reporter: Dhruve Ashar
• Shepherd: Thomas Graves
• Created: 28/Jun/17 14:03
• Updated: 03/Aug/18 07:44
• Resolved: 19/Jul/17 21:00

Description

Right now spark can limit the # of parallel fetches and also limits the amount of data in one fetch, but one fetch to a host could be for 100's of blocks. In one instance we saw 450+ blocks. When you have 100's of those and 1000's of reducers fetching that becomes a lot of metadata and can run the Node Manager out of memory. We should add a config to limit the # of maps per fetch to reduce the load on the NM.

Issue Links

• is duplicated by: SPARK-24989 BlockFetcher should retry while getting OutOfDirectMemoryError
• links to: [Github] Pull Request #18487 (dhruve)
• links to: [Github] Pull Request #18691 (dhruve)