Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28169

Spark can’t push down partition predicate for OR expression

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.1.0
    • Component/s: SQL
    • Labels:

      Description

      Spark can't push down filter condition of Or:

      Such as if I have a table default.test, his partition col is "dt",

      if I use query : 

      select * from default.test where dt=20190625 or (dt = 20190626 and id in (1,2,3) )
      

      In this case, Spark will resolve or condition as one expression, and since this expr has reference of "id", then it can't been push down.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                angerszhuuu angerszhu
                Reporter:
                angerszhuuu angerszhu
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: