[ARROW-462] [C++] Implement in-memory conversions between non-nested primitive types and DictionaryArray equivalent

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: C++
• External issue URL: https://github.com/apache/arrow/issues/16106
• Assignee: Uwe Korn
• Reporter: Wes McKinney
• Created: 06/Jan/17 16:20
• Updated: 11/Jan/23 07:08
• Resolved: 07/Jul/17 00:18

Description

We use a hash table to extract unique values and dictionary indices. There may be an opportunity to consolidate common code from the dictionary encoding implementation implemented in parquet-cpp (but the dictionary indices will not be run-length encoded in Arrow):

https://github.com/apache/parquet-cpp/blob/master/src/parquet/encodings/dictionary-encoding.h

This functionality also needs to permit encoding split across multiple record batches – so the hash table would be a stateful entity, and we can continue to hash more chunks of data to dictionary-encode multiple arrays with a shared dictionary at the end.