[ARROW-6308] [Java] Support write interleaved dictionaries and batches in IPC stream

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Component/s: Java
• External issue URL: https://github.com/apache/arrow/issues/22689
• Assignee: Ji Liu
• Reporter: Ji Liu
• Created: 21/Aug/19 13:58
• Updated: 11/Jan/23 07:46
• Resolved: 28/Jun/20 06:05

Description

Per discussions in the following threads, as spec(http://arrow.apache.org/docs/format/IPC.html#streaming-format) described, as long as a record batch doesn't reference a dictionary they can be interleaved.

https://github.com/apache/arrow/pull/4960

https://github.com/apache/arrow/pull/5146

Currently it’s able to parse dictionaries and batches which are interleaved via ARROW-6040,  But it’s impossible to write data in this format.

cases below should be supported:

i. have a record batch of one dictionary encoded column S

1. Schema
2. RecordBatch: S=[null, null, null, null]
3. DictionaryBatch: ['abc', 'efg']
4. Recordbatch: S=[0, 1, 0, 1]

ii. have a record batch of two dictionary encoded column S1, S2

1. Schema
2. DictionaryBatch S1: ['ab', 'cd']
3. RecordBatch: S1 = [0,1,0,1] S2 =[null, null, null,]
4. DictionaryBatch S2: ['cc', 'dd']
5. RecordBatch: S1 = [0,1,0,1] S2 =[0,1,0,1]

This issue is used to record this problem, and should be done after a ML discuss.