[ARROW-3408] [C++] Add option to CSV reader to dictionary encode individual columns or all string / binary columns

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Component/s: C++
• Labels: csv dataset pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/19735
• Assignee: Antoine Pitrou
• Reporter: Wes McKinney
• Created: 02/Oct/18 17:48
• Updated: 11/Jan/23 07:27
• Resolved: 07/Nov/19 16:49

Description

For many datasets, dictionary encoding everything can result in drastically lower memory usage and subsequently better performance in doing analytics

One difficulty of dictionary encoding in multithreaded conversions is that ideally you end up with one dictionary at the end. So you have two options:

• Implement a concurrent hashing scheme – for low cardinality dictionaries, the overhead associated with mutex contention will not be meaningful, for high cardinality it can be more of a problem

• Hash each chunk separately, then normalize at the end

My guess is that a crude concurrent hash table with a mutex to protect mutations and resizes is going to outperform the latter

Issue Links

• depends upon: ARROW-5052 [C++] Add an incomplete dictionary type
• links to: GitHub Pull Request #5785