Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-2632

[Java] ArrowStreamWriter accumulates ArrowBlock but does not use them

    XMLWordPrintableJSON

    Details

      Description

      ArrowStreamWriter inherits from ArrowWriter the behavior of accumulating an ArrowBlock for each ArrowRecordBatch that is written. But this data is never used in the context of ArrowStreamWriter.

      public abstract class ArrowWriter implements AutoCloseable {
        private final List<ArrowBlock> recordBlocks = new ArrayList<>();
        ...
      
      
         protected void writeRecordBatch(ArrowRecordBatch batch)  {
            ...
           recordBlocks.add(block);
         }
      }
      

      writeRecordBatch can be overridden in ArrowStreamWriter to avoid adding the memory overhead.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                teddy.choi Teddy Choi
                Reporter:
                ewohlstadter Eric Wohlstadter
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 40m
                  1h 40m