[SPARK-27780] Shuffle server & client should be versioned to enable smoother upgrade

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 3.2.0
• Assignee: Unassigned
• Reporter: Imran Rashid
• Created: 20/May/19 16:19
• Updated: 12/Dec/22 18:10

Description

The external shuffle service is often upgraded at a different time than spark itself. However, this causes problems when the protocol changes between the shuffle service and the spark runtime – this forces users to upgrade everything simultaneously.

We should add versioning to the shuffle client & server, so they know what messages the other will support. This would allow better handling of mixed versions, from better error msgs to allowing some mismatched versions (with reduced capabilities).

This originally came up in a discussion here: https://github.com/apache/spark/pull/24565#issuecomment-493496466

There are a few ways we could do the versioning which we still need to discuss:

1) Version specified by config. This allows for mixed versions across the cluster and rolling upgrades. It also will let a spark 3.0 client talk to a 2.4 shuffle service. But, may be a nuisance for users to get this right.

2) Auto-detection during registration with local shuffle service. This makes the versioning easy for the end user, and can even handle a 2.4 shuffle service though it does not support the new versioning. However, it will not handle a rolling upgrade correctly – if the local shuffle service has been upgraded, but other nodes in the cluster have not, it will get the version wrong.

3) Exchange versions per-connection. When a connection is opened, the server & client could first exchange messages with their versions, so they know how to continue communication after that.

Issue Links

• blocks: SPARK-27665 Split fetch shuffle blocks protocol from OpenBlocks