Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-5212

[Go] Array BinaryBuilder in Go library has no access to resize the values buffer

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: Go

      Description

      When you are dealing with a binary builder, there are three buffers: the null bitmap, the offset indexes, and the values buffer which contains the actual data.

      When Reserve or Resize are used, the null bitmap and the offsets are modified to allow for additional appends to function. This seems correct to me. There's no way to know how much the values buffer should be resized until the values are being appended with just the number of values alone.

      But, when you are then appending a bunch of string values, there's no additional API to preallocate the size of that last buffer. That means that batch appending a large amount of strings will constantly allocate even if you know the size ahead of time.

      There should be some additional API to modify this last buffer such as maybe ReserveBytes and ResizeBytes that would correspond with the Reserve and Resize methods, but would related to the values buffer.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jsternberg Jonathan A Sternberg
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m