[ARROW-5767] [Format] Permit dictionary replacements in IPC protocol

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 0.16.0
• Component/s: Format
• External issue URL: https://github.com/apache/arrow/issues/22191
• Assignee: Unassigned
• Reporter: Wes McKinney
• Created: 27/Jun/19 23:35
• Updated: 11/Jan/23 07:42
• Resolved: 08/Jan/20 22:17

Description

We permit dictionaries to grow using the isDelta property in the IPC protocol. I think it should be allowed for the same dictionary ID to appear in an IPC protocol stream but with isDelta=false. This would indicate that the dictionary in that message is to replace any prior-observed ones in subsequent record batches.

For example, we might have dictionary batches in a stream:

id: 0 isDelta: false values: [a, b, c]
id: 0 isDelta: true values [d]
id 0 isDelta: false values [c, a, b]

Such data could easily be produced by a stream producer that is creating dictionaries in different execution threads.

Issue Links

• is duplicated by: ARROW-7283 [Integration] Ensure dictionary IPC implementations match spec clarifications