[ARROW-10406] [C++] Unify dictionaries when writing IPC file in a single shot

• Type: Wish
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26388
• Assignee: Antoine Pitrou
• Reporter: Neal Richardson
• Created: 27/Oct/20 19:10
• Updated: 11/Jan/23 08:13
• Resolved: 09/Feb/21 17:50

Description

I read a big (taxi) csv file and specified that I wanted to dictionary-encode some columns. The resulting Table has ChunkedArrays with 1604 chunks. When I go to write this Table to the IPC file format (write_feather), I get an error:

  Invalid: Dictionary replacement detected when writing IPC file format. Arrow IPC files only support a single dictionary for a given field accross all batches.

I can write this to Parquet and read it back in, and the roundtrip of the data is correct. We should be able to do this in IPC too.

Issue Links

• links to: GitHub Pull Request #9348