Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-2457

garrow_array_builder_append_values() won't work for large arrays

    Details

      Description

      I am using garrow_array_builder_append_values() to transform a native C array to an Arrow array, without calling arrow_array_builder_append multiple times. When calling garrow_array_builder_append_values() in array-builder.cpp with following signature:

      garrow_array_builder_append_values(GArrowArrayBuilder *builder,
      const VALUE *values,
      gint64 values_length,
      const gboolean *is_valids,
      gint64 is_valids_length,
      GError **error,
      const gchar *context)
      

      it will fail for large arrays. This is probably happening because the is_valids array is copied to the valid_bytes array (of different type), for which the memory is allocated on the stack, and not on the heap, like shown on the snippet below:

      uint8_t valid_bytes[is_valids_length];
      for (gint64 i = 0; i < is_valids_length; ++i){ 
        valid_bytes[i] = is_valids[i]; 
      }
      

       A way to avoid this problem would be to allocate memory for the valid_bytes array using malloc() or something similar. Is this behavior intended, maybe because no large arrays should be handed over to that function, or it is rather a bug?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kou Sutou Kouhei
                Reporter:
                harry_g Haralampos Gavriilidis
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: