Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6259

Support parquet filter push down for complex types

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.13.0
    • Fix Version/s: 1.14.0
    • Component/s: None
    • Labels:

      Description

      Currently parquet filter push down is not working for complex types (including arrays).

      This Jira aims to implement filter push down for complex types which underneath type is among supported simple types for filter push down. For instance, currently Drill does not support filter push down for varchars, decimals etc. Though once Drill will start support, this support will be applied for complex type automatically.

      Complex fields will be pushed down the same way regular fields are, except for one case with arrays.

      Query with predicate where users.hobbies_ids[2] is null won't be able to push down because we are not able to determine exact number of nulls in arrays fields. 

      Consider [1, 2, 3] vs [1, 2] if these arrays are in different files. Statistics for the second case won't show any nulls but when querying from two files, in terms of data the third value in array is null.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                arina Arina Ielchiieva
                Reporter:
                arina Arina Ielchiieva
                Reviewer:
                Parth Chandra
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: