Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6054

Parquet dictionary pages should be freed on dictionary construction

    Details

      Description

      The Parquet scanner uses the dictionary_pool_ to allocate memory for the dictionary page (see BaseScalarColumnReader::InitDictionary()). This dictionary page is used to initialize the dictionary in CreateDictionaryDecoder(). The resulting dictionary is a vector of values. For some datatypes, such as strings, the resulting dictionary has an array of StringValue's that contain pointers into the dictionary page (see the StringValue specialization in ParquetPlainEncoder::Decode()). In this case, the dictionary page must be kept and attached to the last row batch that references it. However, for other datatypes, the values are copied into the dictionary and the dictionary page is no longer needed after the dictionary is constructed.

      Currently, these dictionary pages remain in the dictionary_pool_ and are attached to the last row batch to be passed to other ExecNodes (see FlushRowGroupResources()). This should only pass StringValue dictionary pages (or other types that point to data in the page) on the row batch. The other types should be freed immediately once the dictionary has been constructed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                csringhofer Csaba Ringhofer
                Reporter:
                joemcdonnell Joe McDonnell
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: