[SPARK-6166] Limit number of in flight outbound requests for shuffle fetch

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Assignee: Sanket Reddy
• Reporter: Mridul Muralidharan
• Created: 04/Mar/15 21:27
• Updated: 12/Feb/16 22:15
• Resolved: 12/Feb/16 06:41

Description

spark.reducer.maxMbInFlight puts a bound on the in flight data in terms of size.
But this is not always sufficient : when the number of hosts in the cluster increase, this can lead to very large number of in-bound connections to one more nodes - causing workers to fail under the load.

I propose we also add a spark.reducer.maxReqsInFlight - which puts a bound on number of outstanding outbound requests.
This might still cause hotspots in the cluster, but in our tests this has significantly reduced the occurance of worker failures.

Issue Links

• links to: [Github] Pull Request #5852 (mridulm)
• links to: [Github] Pull Request #10838 (redsanket)