[SPARK-48610] Remove ExplainUtils.processPlan synchronize

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0, 3.5.2
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Ziqi Liu
• Reporter: Ziqi Liu
• Created: 13/Jun/24 00:45
• Updated: 10/Oct/24 00:04
• Resolved: 17/Jun/24 06:35

Description

https://github.com/apache/spark/pull/45282 introduced synchronize to `ExplainUtils.processPlan`  to avoid race condition when multiple queries refers to same cached plan.

The granularity of lock is too large. We can try to fix the root cause of this concurrency issue by refactoring the usage of mutable OP_ID_TAG, which is not a good practice in terms of immutable nature of SparkPlan.  Instead, we can use an auxiliary id map, with object identity as the key.

Issue Links

• links to: GitHub Pull Request #7445
• links to: GitHub Pull Request #46965