Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
3.1.0, 3.0.0, 3.1.1, 3.1.2
Description
Complex data types like MAP, STRUCT cannot be serialized/deserialzed using Arrow SerDe when there are more than 1024 values. This happens due to ColumnVector always being initialized with a size of 1024.
Sample unit test to reproduce the case in TestArrowColumnarBatchSerDe :
@Test public void testListBooleanWithMoreThan1024Values() throws SerDeException { String[][] schema = { {"boolean_list", "array<boolean>"}, }; Object[][] rows = new Object[1025][1]; for (int i = 0; i < 1025; i++) { rows[i][0] = new BooleanWritable(true); } initAndSerializeAndDeserialize(schema, toList(rows)); }
Attachments
Issue Links
- links to