[ARROW-3144] [C++] Move "dictionary" member from DictionaryType to ArrayData to allow for changing dictionaries between Array chunks

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/19492
• Assignee: Wes McKinney
• Reporter: Wes McKinney
• Created: 29/Aug/18 21:14
• Updated: 11/Jan/23 07:25
• Resolved: 17/May/19 16:41

Description

There are a couple of inter-related issues:

• Cases where a system might send the schema without the dictionaries, and the user wishes to reason about the schema and its types without knowing the dictionary values

• Dictionaries that are changing, e.g. using delta dictionary messages

arrow::DictionaryType has no "linkage" to any external object. I propose adding a "LinkedDictionaryType" or something similar (purely a C++ construct), which functionally would be a subclass of DictionaryType, which would allow a type to be created which will obtain its dictionary later through some kind of "Dictionary provider" interface. There is something similar in Java already. This would allow a dictionary to evolve via delta dictionaries, or for a dictionary to be retrieved later e.g. through an RPC or IPC layer

Issue Links

• blocks: ARROW-1278 Integration tests for Fixed Size List type
• causes: ARROW-5361 [R] Follow DictionaryType/DictionaryArray changes from ARROW-3144
• is depended upon by: ARROW-5279 [C++] Support reading delta dictionaries in IPC streams
• is depended upon by: ARROW-3325 [Python] Support reading Parquet binary/string columns directly as DictionaryArray
• is depended upon by: ARROW-5336 [C++] Implement arrow::Concatenate for dictionary-encoded arrays with unequal dictionaries
• is related to: ARROW-5338 [Format][Integration] Define how to test for delta dictionary support in the JSON integration test data format
• links to: GitHub Pull Request #4316