[SPARK-42252] Deprecate spark.shuffle.unsafe.file.output.buffer and add a new config

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.1.0, 3.2.0, 3.3.0, 3.4.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Wei Guo
• Reporter: Wei Guo
• Created: 31/Jan/23 04:38
• Updated: 14/Jun/24 07:12
• Resolved: 14/Jun/24 07:12

Description

After Jira SPARK-28209 and PR [25007|https://github.com/apache/spark/pull/25007], the new shuffle writer api is proposed. All shuffle writers(BypassMergeSortShuffleWriter, SortShuffleWriter, UnsafeShuffleWriter) are based on LocalDiskShuffleMapOutputWriter to write local disk shuffle files. The config spark.shuffle.unsafe.file.output.buffer used in LocalDiskShuffleMapOutputWriter was only used in UnsafeShuffleWriter before.
 
It's better to rename it and make it more suitable.

Issue Links

• links to: [Github] Pull Request #39819 (wayneguow)
• links to: [Github] Pull Request #39819 (wayneguow)
• links to: GitHub Pull Request #39819