Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12957

Derive and propagate data constrains in logical plan

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • SQL
    • None

    Description

      Based on the semantic of a query plan, we can derive data constrains (e.g. if a filter defines a > 10, we know that the output data of this filter satisfy the constrain of a > 10 and a is not null). We should build a framework to derive and propagate constrains in the logical plan, which can help us to build more advanced optimizations.

      Attachments

        1. ConstraintPropagationinSparkSQL.pdf
          199 kB
          Sameer Agarwal

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-19609 Broadcast joins should pushdown join constraints as Filter to the larger relation

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #10844 [Github] Pull Request #10844 (sameeragarwal)

          There are no Sub-Tasks for this issue.

          Activity

            People

              sameerag Sameer Agarwal
              yhuai Yin Huai
              Votes:
              2 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: