[ARROW-12101] [Format] Consider adding int0 and other small integer types for more efficient Dictionary encoding

• Type: Wish
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Format
• External issue URL: https://github.com/apache/arrow/issues/27925
• Assignee: Unassigned
• Reporter: Justin Talbot
• Created: 26/Mar/21 18:35
• Updated: 11/Jan/23 08:24

Description

I often come across the need to store single-valued columns. The current Arrow format doesn't have an efficient way to represent these, I believe. One possible improvement would be to introduce an int0 type (where all values are 0) that, like null, does not have a buffer allocated. Then this could be used as an index into a Dictionary with a single value.

For low cardinality columns, I also often find myself wishing for int1, int2, and int4 types to use as an index.