[SPARK-44345] Lower unknown shuffle map output log level to WARN if shuffle migration is enabled

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Zhongwei Zhu
• Reporter: Zhongwei Zhu
• Created: 09/Jul/23 04:47
• Updated: 24/Sep/23 02:04
• Resolved: 24/Sep/23 02:03

Description

When decommission and shuffle migration is enabled, there're lots of error message like `Asked to update map output for unknown shuffle` .

As shuffle clean and unregister is done by ContextCleaner in an async way, when target block manager received the shuffle block from decommissioned block manager, then update shuffle location to map output tracker, but at that time, shuffle might have been unregistered. This should not consider as error.

Issue Links

• links to: GitHub Pull Request #41906