[SPARK-41048] Improve output partitioning and ordering with AQE cache

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: XiDuo You
• Reporter: XiDuo You
• Created: 08/Nov/22 10:47
• Updated: 22/Sep/23 18:40
• Resolved: 17/Nov/22 03:35

Description

The cached plan in InMemoryRelation can be AdaptiveSparkPlanExec, however AdaptiveSparkPlanExec deos not specify its output partitioning and ordering. It causes unnecessary shuffle and local sort for downstream action.

 

Issue Links

• causes: SPARK-45282 Join loses records for cached datasets
• links to: [Github] Pull Request #38558 (ulysses-you)
• links to: [Github] Pull Request #38558 (ulysses-you)