Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-751

DictionaryFilter patch broke column projection

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.9.0
    • Fix Version/s: 1.9.0, 1.8.2
    • Component/s: parquet-mr
    • Labels:
      None

      Description

      The DictionaryFilter patch included a change to reuse a ParquetFileReader rather than creating new ones (and opening new file streams). But by reusing a reader, the projection columns are no longer set. The result is that all columns are read from disk, but only the requested columns are materialized. The fix is to set the requested schema on the file reader when it is fetched from the read context inside InternalParquetRecordReader.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rdblue Ryan Blue
                Reporter:
                rdblue Ryan Blue
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: