Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24802

Optimization Rule Exclusion

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.4.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Since Spark has provided fairly clear interfaces for adding user-defined optimization rules, it would be nice to have an easy-to-use interface for excluding an optimization rule from the Spark query optimizer as well.

      This would make customizing Spark optimizer easier and sometimes could debugging issues too.

      1. Add a new config spark.sql.optimizer.excludedRules, with the value being a list of rule names separated by comma.
      2. Modify the current batches method to remove the excluded rules from the default batches. Log the rules that have been excluded.
      3. Split the existing default batches into "post-analysis batches" and "optimization batches" so that only rules in the "optimization batches" can be excluded.

        Attachments

          Activity

            People

            • Assignee:
              maryannxue Wei Xue
              Reporter:
              maryannxue Wei Xue
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: