Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19290

add a new extending interface in Analyzer for post-hoc resolution

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.2.0
    • SQL
    • None

    Description

      To implement DDL commands, we added several analyzer rules in sql/hive module to analyze DDL related plans. However, our Analyzer currently only have one extending interface: extendedResolutionRules, which defines extra rules that will be run together with other rules in the resolution batch, and doesn't fit DDL rules well, because:

      1. DDL rules may do some checking and normalization, but we may do it many times as the resolution batch will run rules again and again, until fixed point, and it's hard to tell if a DDL rule has already done its checking and normalization. It's fine because DDL rules are idempotent, but it's bad for analysis performance
      2. some DDL rules may depend on others, and it's pretty hard to write if conditions to guarantee the dependencies. It will be good if we have a batch which run rules in one pass, so that we can guarantee the dependencies by rules order.

      Attachments

        Activity

          People

            cloud_fan Wenchen Fan
            cloud_fan Wenchen Fan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: