[ARROW-3325] [Python] Support reading Parquet binary/string columns directly as DictionaryArray

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15.0
• Component/s: Python
• Labels: parquet pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/19660
• Assignee: Wes McKinney
• Reporter: Wes McKinney
• Created: 25/Sep/18 11:23
• Updated: 11/Jan/23 07:26
• Resolved: 05/Aug/19 18:12

Description

Requires PARQUET-1324 and probably quite a bit of extra work

Properly implementing this will require dictionary normalization across row groups. When reading a new row group, a fast path that compares the current dictionary with the prior dictionary should be used. This also needs to handle the case where a column chunk "fell back" to PLAIN encoding mid-stream

Issue Links

• depends upon: ARROW-3144 [C++] Move "dictionary" member from DictionaryType to ArrayData to allow for changing dictionaries between Array chunks
• depends upon: ARROW-3772 [C++] Read Parquet dictionary encoded ColumnChunks directly into an Arrow DictionaryArray
• depends upon: ARROW-6042 [C++] Implement alternative DictionaryBuilder that always yields int32 indices
• depends upon: ARROW-6065 [C++] Reorganize parquet/arrow/reader.cc, remove code duplication, improve readability
• depends upon: ARROW-6077 [C++][Parquet] Build logical schema tree mapping Arrow fields to Parquet schema levels
• is related to: ARROW-3652 [Python] CategoricalIndex is lost after reading back
• relates to: ARROW-6140 [C++][Parquet] Support direct dictionary decoding of types other than BYTE_ARRAY
• relates to: ARROW-3769 [C++] Support reading non-dictionary encoded binary Parquet columns directly as DictionaryArray
• relates to: ARROW-3772 [C++] Read Parquet dictionary encoded ColumnChunks directly into an Arrow DictionaryArray
• relates to: ARROW-3246 [Python][Parquet] direct reading/writing of pandas categoricals in parquet
• links to: GitHub Pull Request #4999
• links to: GitHub Pull Request #5018