[SPARK-44953] Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: binjie yang
• Reporter: Holden Karau
• Created: 24/Aug/23 19:03
• Updated: 13/May/24 20:38
• Resolved: 13/May/24 20:37

Description

Some people enable both shuffle tracking and another mechanism (like migration) and then are confused when their jobs don't scale down.

 

We should at least log a warning here (or automatically disable shuffle tracking?) when it is configured alongside another DA supported mechanism.

Issue Links

• links to: GitHub Pull Request #45454