[SPARK-45792] SPIP: ShuffleManager short name registration via SparkPlugin

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: Unassigned
• Reporter: Alessandro Bellina
• Created: 04/Nov/23 21:33
• Updated: 07/Jun/24 06:10

Description

We would like to make it possible for a SparkPlugin to expose custom ShuffleManager implementations using short names, in addition to default configurations for each short name in order to improve ease of use. 

Today, users leveraging a spark plugin need to set a class name under spark.plugins and if the plugin also provides a ShuffleManager, the user needs to set spark.shuffle.manager to a fully qualified class name. Additionally, users need to make sure they have set Spark configurations that are required or recommended for the shuffle manager instance. This can be cumbersome for the user, adding barriers to the pluggable interface that Spark provides.

Spark provides a short name for SortShuffleManager (“sort”) today. The idea in this SPIP is to make this set of short names extensible by a SparkPlugin.

SPIP: https://docs.google.com/document/d/1flijDjMMAAGh2C2k-vg1u651RItaRquLGB_sVudxf6I/edit?usp=sharing

Issue Links

• is blocked by: SPARK-45762 Shuffle managers defined in user jars are not available for some launch modes
• links to: GitHub Pull Request #6013