Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21218

Convert IN predicate to equivalent Parquet filter

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.1.1
    • None
    • Optimizer, SQL
    • None

    Description

      Convert IN predicate to equivalent expression involving equality conditions to allow the filter to be pushed down to Parquet.

      For instance,

      C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20)

      Attachments

        1. OR Predicate.png
          27 kB
          Michael Styles
        2. IN Predicate.png
          27 kB
          Michael Styles

        Issue Links

          duplicates

          Sub-task - The sub-task of the issue SPARK-17091 Convert IN predicate to equivalent Parquet filter

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #18424 [Github] Pull Request #18424 (ptkool)

          Activity

            People

              Unassigned Unassigned
              ptkool Michael Styles
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: