[ARROW-8464] [Rust] [DataFusion] Add better and faster support for dictionary types

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Component/s: Rust - DataFusion
• External issue URL: https://github.com/apache/arrow/issues/24639
• Assignee: Unassigned
• Reporter: Andy Grove
• Created: 14/Apr/20 22:54
• Updated: 11/Jan/23 08:00
• Resolved: 26/Apr/21 13:17

Description

Usecases: Efficiently process large columns of low cardinality Strings
 

• BatchIterator should accept both DictionaryBatch and RecordBatch
• Type Coercion optimizer rule should inject expression for converting dictionary value types to index types (for equality expressions, and IN(values, ...)
• Physical expression would lookup index for dictionary values referenced in the query so that at runtime, only indices are being compared per batch

Issue Links

• relates to: ARROW-10159 [Rust][DataFusion] Add support for Dictionary types in data fusion