Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33828

SQL Adaptive Query Execution QA

    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 3.2.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      Since SPARK-31412 is delivered at 3.0.0, we received and handled many JIRA issues at 3.0.x/3.1.0/3.2.0. This umbrella JIRA issue aims to enable it by default and collect all information in order to do QA for this feature in Apache Spark 3.2.0 timeframe.

        Attachments

          Issue Links

          1.
          Enable spark.sql.adaptive.enabled true by default Sub-task Resolved Dongjoon Hyun
          2.
          support Dynamic Partition Pruning in Adaptive Execution Sub-task Open Unassigned
          3.
          TPCDS Q5 fails if spark.sql.adaptive.enabled=true Sub-task Resolved Takeshi Yamamuro
          4.
          Use the `CastSupport.cast` method in HashJoin Sub-task Closed Unassigned
          5.
          EXPLAIN FORMATTED doesn't show the plan for subqueries if AQE is enabled Sub-task Resolved Kousuke Saruta
          6.
          Fix some tests with AQE enabled Sub-task Resolved wuyi
          7.
          Broadcast timeout happened unexpectedly in AQE Sub-task Resolved Yu Zhong
          8.
          Add an option in AQE to mitigate skew even if it causes an new shuffle Sub-task In Progress Unassigned
          9.
          Support DPP in AQE When the join is Broadcast hash join before applying the AQE rules Sub-task Resolved Ke Jia
          10.
          Regression in "operating on canonicalized plan" check in CustomShuffleReaderExec Sub-task Resolved Andy Grove
          11.
          Fail in fetch shuffle blocks in batch when i/o encryption is enabled. Sub-task Resolved hezuojiao
          12.
          Support fetching shuffle blocks in batch with i/o encryption Sub-task Open Unassigned
          13.
          Support DPP in AQE when the boradcast exchange can be reused Sub-task In Progress Unassigned
          14.
          AQE columnar mismatch on exchange reuse Sub-task In Progress Unassigned
          15.
          mapred.reduce.tasks should be shuffle.partitions not adaptive.coalescePartitions.initialPartitionNum Sub-task Resolved Kent Yao
          16.
          Support AQE side broadcastJoin threshold Sub-task Resolved XiDuo You
          17.
          Use origin plan if can not coalesce shuffle partition Sub-task Resolved XiDuo You
          18.
          Coalesce shuffle partition should handle empty input RDD Sub-task Resolved XiDuo You
          19.
          OptimizeSkewedJoin support ShuffledHashJoinExec Sub-task Resolved XiDuo You
          20.
          Support AQE side shuffled hash join formula Sub-task In Progress Unassigned
          21.
          Support coalesce partition through union Sub-task In Progress Unassigned
          22.
          EXPLAIN CODEGEN does not work with AQE Sub-task Resolved Cheng Su

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                dongjoon Dongjoon Hyun
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated: