[SPARK-33828] SQL Adaptive Query Execution QA

• Type: Umbrella
• Status: Closed
• Priority: Critical
• Resolution: Done
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Labels: releasenotes
• Target Version/s: 3.2.0
• Assignee: Unassigned Assign to me
• Reporter: Dongjoon Hyun
• Created: 17/Dec/20 18:11
• Updated: 27/Jan/22 20:06
• Resolved: 19/Oct/21 22:03

Description

Since SPARK-31412 is delivered at 3.0.0, we received and handled many JIRA issues at 3.0.x/3.1.0/3.2.0. This umbrella JIRA issue aims to enable it by default and collect all information in order to do QA for this feature in Apache Spark 3.2.0 timeframe.

Issue Links

• is related to: SPARK-37063 SQL Adaptive Query Execution QA: Phase 2
• relates to: SPARK-31412 New Adaptive Query Execution in Spark SQL

Sub-Tasks

• Enable spark.sql.adaptive.enabled true by default: Resolved
• support Dynamic Partition Pruning in Adaptive Execution: Resolved
• TPCDS Q5 fails if spark.sql.adaptive.enabled=true: Resolved
• Use the `CastSupport.cast` method in HashJoin: Closed
• EXPLAIN FORMATTED doesn't show the plan for subqueries if AQE is enabled: Resolved
• Fix some tests with AQE enabled: Resolved
• Broadcast timeout happened unexpectedly in AQE: Resolved
• Support DPP in AQE When the join is Broadcast hash join before applying the AQE rules: Resolved
• Regression in "operating on canonicalized plan" check in CustomShuffleReaderExec: Resolved
• Fail in fetch shuffle blocks in batch when i/o encryption is enabled.: Resolved
• Support DPP in AQE when the broadcast exchange can be reused: Resolved
• AQE columnar mismatch on exchange reuse: Resolved
• Disable timeout for BroadcastQueryStageExec in AQE: Resolved
• mapred.reduce.tasks should be shuffle.partitions not adaptive.coalescePartitions.initialPartitionNum: Resolved
• Support AQE side broadcastJoin threshold: Resolved
• Support AQE optimizer side transformUpWithPruning: Resolved
• Use origin plan if can not coalesce shuffle partition: Resolved
• Use inputPlan instead of currentPhysicalPlan to initialize logical link: Resolved
• Coalesce shuffle partition should handle empty input RDD: Resolved
• OptimizeSkewedJoin support ShuffledHashJoinExec: Resolved
• Support AQE side shuffled hash join formula: Resolved
• EXPLAIN CODEGEN does not work with AQE: Resolved
• Do not remove REPARTITION_BY_NUM shuffle if AQE is enabled: Resolved
• Enhance EliminateUnnecessaryJoin: Resolved
• UnsupportedOperationException: WholeStageCodegen (3) does not implement doExecuteBroadcast: Resolved
• QueryExecutionListener does not see any observed metrics fired before persist/cache: Resolved
• Add dataSize field in CoalescedPartitionSpec: Resolved
• EXPLAIN FORMATTED for AQE: Resolved
• Make sure partitions are not too small in AQE partition coalescing: Resolved
• Use ValidateRequirements instead of EnsureRequirements to skip AQE rules that adds extra shuffles: Resolved
• [SQL] Regression in AQEShuffleReadExec: Resolved
• AQE Shuffle should wait for its subqueries to finish before materializing: Resolved
• Adaptively enable timeout for BroadcastQueryStageExec: In Progress