Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-389

Filter predicates should work with missing columns

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0, 1.7.0, 1.8.0
    • Fix Version/s: 1.9.0, 1.8.2
    • Component/s: parquet-mr
    • Labels:
      None

      Description

      This issue originates from SPARK-11103, which contains detailed information about how to reproduce it.

      The major problem here is that, filter predicates pushed down assert that columns they touch must exist in the target physical files. But this isn't true in case of schema merging.

      Actually this assertion is unnecessary, because if a column is missing in the filter schema, the column is considered to be filled by nulls, and all the filters should be able to act accordingly. For example, if we push down a = 1 but a is missing in the underlying physical file, all records in this file should be dropped since a is always null. On the other hand, if we push down a IS NULL, all records should be preserved.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rdblue Ryan Blue
                Reporter:
                lian cheng Cheng Lian
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: