[ARROW-3772] [C++] Read Parquet dictionary encoded ColumnChunks directly into an Arrow DictionaryArray

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15.0
• Component/s: C++
• Labels: parquet pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/20110
• Assignee: Wes McKinney
• Reporter: Stav Nir
• Created: 13/Jun/18 14:08
• Updated: 11/Jan/23 07:29
• Resolved: 26/Jul/19 23:54

Description

Dictionary data is very common in parquet, in the current implementation parquet-cpp decodes dictionary encoded data always before creating a plain arrow array. This process is wasteful since we could use arrow's DictionaryArray directly and achieve several benefits:

1. Smaller memory footprint - both in the decoding process and in the resulting arrow table - especially when the dict values are large
2. Better decoding performance - mostly as a result of the first bullet - less memory fetches and less allocations.

I think those benefits could achieve significant improvements in runtime.

My direction for the implementation is to read the indices (through the DictionaryDecoder, after the RLE decoding) and values separately into 2 arrays and create a DictionaryArray using them.

There are some questions to discuss:

1. Should this be the default behavior for dictionary encoded data
2. Should it be controlled with a parameter in the API
3. What should be the policy in case some of the chunks are dictionary encoded and some are not.

I started implementing this but would like to hear your opinions.

Issue Links

• is depended upon by: ARROW-3325 [Python] Support reading Parquet binary/string columns directly as DictionaryArray
• is related to: ARROW-3652 [Python] CategoricalIndex is lost after reading back
• is related to: ARROW-5993 [Python] Reading a dictionary column from Parquet results in disproportionate memory usage
• is related to: ARROW-5984 [C++] Provide method on AdaptiveIntBuilder for appending integer Array types
• is related to: ARROW-3325 [Python] Support reading Parquet binary/string columns directly as DictionaryArray
• is related to: ARROW-6049 [C++] Support using Array::View from compatible dictionary type to another
• is related to: ARROW-3246 [Python][Parquet] direct reading/writing of pandas categoricals in parquet
• relates to: ARROW-6140 [C++][Parquet] Support direct dictionary decoding of types other than BYTE_ARRAY
• links to: GitHub Pull Request #4949