[SPARK-25641] Change the spark.shuffle.server.chunkFetchHandlerThreadsPercent default to 100

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Sanket Reddy
• Reporter: Sanket Reddy
• Created: 04/Oct/18 14:48
• Updated: 27/Mar/20 03:02
• Resolved: 08/Oct/18 18:20

Description

We want to change the default percentage to 100 for spark.shuffle.server.chunkFetchHandlerThreadsPercent. The reason being
currently this is set to 0. Which means currently if server.ioThreads > 0, the default number of threads would be 2 * #cores instead of server.io.Threads. We want the default to server.io.Threads in case this is not set at all. Also here a default of 0 would also mean 2 * #cores

Issue Links

• relates to: SPARK-24355 Improve Spark shuffle server responsiveness to non-ChunkFetch requests
• links to: [Github] Pull Request #22628 (redsanket)