Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26345 Parquet support Column indexes
  3. SPARK-33673

Do not push down partition filters to ParquetScan for DataSourceV2

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • SQL
    • None

    Description

      - Spark vectorized reader - with partition data column - select a single complex field array and its parent struct array *** FAILED ***
      - Non-vectorized reader - with partition data column - select a single complex field array and its parent struct array *** FAILED ***
      - Spark vectorized reader - with partition data column - select a single complex field from a map entry and its parent map entry *** FAILED ***
      - Non-vectorized reader - with partition data column - select a single complex field from a map entry and its parent map entry *** FAILED ***
      - Spark vectorized reader - with partition data column - partial schema intersection - select missing subfield *** FAILED ***
      - Non-vectorized reader - with partition data column - partial schema intersection - select missing subfield *** FAILED ***
      - Spark vectorized reader - with partition data column - no unnecessary schema pruning *** FAILED ***
      - Non-vectorized reader - with partition data column - no unnecessary schema pruning *** FAILED ***
      - Spark vectorized reader - with partition data column - empty schema intersection *** FAILED ***
      - Non-vectorized reader - with partition data column - empty schema intersection *** FAILED ***
      - Spark vectorized reader - with partition data column - select a single complex field and in where clause *** FAILED ***
      - Non-vectorized reader - with partition data column - select a single complex field and in where clause *** FAILED ***
      - Spark vectorized reader - with partition data column - select nullable complex field and having is not null predicate *** FAILED ***
      - Non-vectorized reader - with partition data column - select nullable complex field and having is not null predicate *** FAILED ***
      

      These test will fail, this is because Parquet will return empty results for non-existent column since PARQUET-1765.

      Attachments

        Activity

          People

            LuciferYang Yang Jie
            yumwang Yuming Wang
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: