[ARROW-7156] [C#] Large record batch is written with negative buffer length

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.14.1, 0.15.1
• Component/s: C#
• External issue URL: https://github.com/apache/arrow/issues/23458
• Assignee: Unassigned
• Reporter: Anthony Abate
• Created: 13/Nov/19 15:12
• Updated: 11/Jan/23 07:51

Description

I have a 30 gig arrow file with 100 batches.  the largest batch in the file causes get batch to fail - All other batches load fine. in 14.11 the individual batch errors.. in 15.1.1 the batch crashes R studio when it is used

14.1.1

>  rbn <- data_rbfr$get_batch(x)
Error in ipc__RecordBatchFileReader_ReadRecordBatch(self, i) : 
Invalid: negative malloc size
  

15.1.1

rbn <- data_rbfr$get_batch(x)  works!
df <- as.data.frame(rbn) - Crashes R Studio! 

 

Update

I put the data in the batch into a separate file.  The file size is over 2 gigs. 

Using 15.1.1, when I try to load this entire file via read_arrow it also fails.

ar <- arrow::read_arrow("e:\\temp\\file.arrow") 
Error in Table__from_RecordBatchFileReader(batch_reader) :
 Invalid: negative malloc size