Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40169

Fix the issue with Parquet column index and predicate pushdown in Data source V1

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.1, 3.2.3, 3.4.0
    • 3.3.1, 3.2.3, 3.4.0
    • SQL
    • None

    Description

      This is a follow for SPARK-39833. In https://github.com/apache/spark/pull/37419, we disabled column index for Parquet due to correctness issues that we found when filtering data on the partition column overlapping with data schema.

       

      This ticket is for permanent and thorough fix for the issue and re-enablement of the column index. See more details in the PR linked above.

      Attachments

        Issue Links

          Activity

            People

              csun Chao Sun
              ivan.sadikov Ivan Sadikov
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: