Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25556 Predicate Pushdown for Nested fields
  3. SPARK-25559

Just remove the unsupported predicates in Parquet

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Currently, in ParquetFilters, if one of the children predicates is not supported by Parquet, the entire predicates will be thrown away. In fact, if the unsupported predicate is in the top level And condition or in the child before hitting Not or Or condition, it's safe to just remove the unsupported one as unhandled filters.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dbtsai DB Tsai
                Reporter:
                dbtsai DB Tsai
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: