[ARROW-2632] [Java] ArrowStreamWriter accumulates ArrowBlock but does not use them

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Component/s: Java
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/19028
• Assignee: Teddy Choi
• Reporter: Eric Wohlstadter
• Created: 23/May/18 17:34
• Updated: 11/Jan/23 07:22
• Resolved: 29/May/18 22:29

Description

ArrowStreamWriter inherits from ArrowWriter the behavior of accumulating an ArrowBlock for each ArrowRecordBatch that is written. But this data is never used in the context of ArrowStreamWriter.

public abstract class ArrowWriter implements AutoCloseable {
  private final List<ArrowBlock> recordBlocks = new ArrayList<>();
  ...


   protected void writeRecordBatch(ArrowRecordBatch batch)  {
      ...
     recordBlocks.add(block);
   }
}

writeRecordBatch can be overridden in ArrowStreamWriter to avoid adding the memory overhead.

Issue Links

• relates to: ARROW-2645 [Java] ArrowStreamWriter accumulates DictionaryBatch ArrowBlocks
• links to: GitHub Pull Request #2079