Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-11364

Use Late Materialization when structs are in the select list

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • Impala 4.1.0
    • None
    • Backend

    Description

      IMPALA-9496 introduced querying structs in the select list even for Parquet tables. However, it has a restriction that in this case late materialization (LM) can't be used.

      The reason is that LM expects the column readers to be used through the batched reading interface, however, currently struct column readers use the non-batched reading interface of its children. As a result after reading, the column readers are not in a state as SkipRows() of LM expects and then results in a query failure because it's not able to skip the rows for non-filter readers.

      Once IMPALA-11363 is implemented and the struct will also use the ReadValueBatch() interface of its children then late materialization could be turned on even if structs are in the select list.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              gaborkaszab Gabor Kaszab
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: