Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37063

SQL Adaptive Query Execution QA: Phase 2

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.3.0
    • None
    • SQL
    • None

    Attachments

      Issue Links

        relates to

        Umbrella - An overarching type made of sub-tasks SPARK-33828 SQL Adaptive Query Execution QA

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed
        1.
        		 Support fetching shuffle blocks in batch with i/o encryption Sub-task In Progress Unassigned
        2.
        		 Completely fix the broadcast timeout issue in AQE Sub-task Resolved Unassigned
        3.
        		 Flaky Test: AdaptiveQueryExecSuite.multiple joins Sub-task Resolved Unassigned
        4.
        		 Make config maxShuffledHashJoinLocalMapThreshold fallback to advisoryPartitionSizeInBytes Sub-task In Progress Unassigned
        5.
        		 Demote BroadcastJoin causes performance regression and increases OOM risks Sub-task Open Unassigned
        6.
        		 Cancel all running job after AQE plan finished Sub-task In Progress Unassigned
        7.
        		 OptimizeSkewInRebalancePartitions support optimize non-root node Sub-task Resolved XiDuo You
        8.
        		 Add small partition factor for rebalance partitions Sub-task Resolved XiDuo You
        9.
        		 Support optimize skewed join through union Sub-task Resolved Unassigned
        10.
        		 In AQE, wrong InMemoryRelation size estimation causes "Cannot broadcast the table that is larger than 8GB: 8 GB" failure Sub-task Resolved Michael Chen
        11.
        		 SPARK-33832 brings the bug that OptimizeSkewedJoin may not work since it was applied on whole plan innstead of new stage plan Sub-task Resolved Wenchen Fan
        12.
        		 Add an option in AQE to mitigate skew even if it causes an new shuffle Sub-task Resolved XiDuo You
        13.
        		 Support coalesce partition through union Sub-task Resolved XiDuo You
        14.
        		 Support eliminate limits in AQE Optimizer Sub-task Resolved XiDuo You
        15.
        		 AQE dynamic join selection should not apply to non-equi join Sub-task Resolved Cheng Su
        16.
        		 AQE can change bhj to smj if no extra shuffle introduce Sub-task Resolved XiDuo You
        17.
        		 Query with cast containing non-nullable columns fails with AQE on Spark 3.1.1 Sub-task Resolved Shardul Mahadik
        18.
        		 Explain formatted does not collect subqueries under query stage in AQE Sub-task Resolved XiDuo You
        19.
        		 Unify get preferred locations for shuffle in AQE Sub-task In Progress Unassigned
        20.
        		 Improve perfermance of ShufflePartitionsUtil createSkewPartitionSpecs Sub-task Resolved XiDuo You
        21.
        		 AQE throw exception should respect SparkFatalException Sub-task Resolved XiDuo You
        22.
        		 TPCDS 1TB q72 fails when spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly is false Sub-task Resolved XiDuo You
        23.
        		 Avoid unnecessary sort in FileFormatWriter if user has specified sort in AQE Sub-task In Progress Unassigned
        24.
        		 Extend SparkSessionExtensions to inject rules into AQE Optimizer Sub-task Resolved XiDuo You
        25.
        		 Optimize one row plan in normal and AQE Optimizer Sub-task Resolved XiDuo You

        Activity

          People

            Unassigned Unassigned
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated: