[ARROW-255] Finalize Dictionary representation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.1.0
• Component/s: Format
• External issue URL: https://github.com/apache/arrow/issues/15596
• Assignee: Julien Le Dem
• Reporter: Julien Le Dem
• Created: 10/Aug/16 22:01
• Updated: 11/Jan/23 07:07
• Resolved: 20/Aug/16 20:03

Description

format/Messages.fbs mentions DictionaryBatches with an id but does not specify where they are referenced.

We should add a dictionary: long in Field that references the dictionary id:

Field: https://github.com/apache/arrow/blob/34e7f48cb71428c4d78cf00d8fdf0045532d6607/format/Message.fbs#L86

Dictionary id: https://github.com/apache/arrow/blob/34e7f48cb71428c4d78cf00d8fdf0045532d6607/format/Message.fbs#L165

We need a spec in format/Layout.md that describes the dictionary layout.
When dictionary encoded the value vector is an array of signed int32 (for consistency with variable length collection offsets).
The dictionary vector is a Vector of the type of the value. indexed by their id in the dictionary.

Issue Links

• links to: PR