[SPARK-38578] Avoid unnecessary sort in FileFormatWriter if user has specified sort in AQE

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: XiDuo You
• Created: 17/Mar/22 02:08
• Updated: 21/Mar/22 13:27

Description

FileFormatWriter will check and add an implicit sort for dynamic partition columns or bucket columns according to the input physical plan. The check became always failure since AQE AdaptiveSparkPlanExec has no outputOrdering.

That casues a redundant sort if user has specified a sort which satisfies the required ordering (dynamic partition and bucket columns).

Issue Links

• links to: [Github] Pull Request #35924 (ulysses-you)