[HIVE-21966] Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 4.0.0-alpha-1
• Component/s: llap, Serializers/Deserializers
• Labels: pull-request-available
• Assignee: Shubham Chaurasia Assign to me
• Reporter: Shubham Chaurasia
• Created: 08/Jul/19 09:20
• Updated: 17/Nov/22 08:50
• Resolved: 09/Jul/19 03:55

Description

When we submit query through llap-ext-client, arrow serializer throws ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.

1) hive.vectorized.execution.filesink.arrow.native.enabled=true to take arrow serializer code path.
2) Query contains a filter or limit clause which enforces VectorizedRowBatch#selectedInUse=true
3) Projection involves a column of type MultiValuedColumnVector.

Sample stacktrace:

Caused by: java.lang.ArrayIndexOutOfBoundsException: 150
	at org.apache.hadoop.hive.ql.io.arrow.Serializer.writeGeneric(Serializer.java:679)
	at org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:518)
	at org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:276)
	at org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:342)
	at org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:282)
	at org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:365)
	at org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:279)
	at org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:199)
	at org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:135)
	... 30 more

It can be reproduced by:

from beeline:

CREATE TABLE complex_tbl(c1 array<struct<f1:string,f2:string>>) STORED AS ORC;
INSERT INTO complex_tbl SELECT ARRAY(NAMED_STRUCT('f1','v11', 'f2','v21'), NAMED_STRUCT('f1','v21', 'f2','v22'));

and when we fire query: select * from complex_tbl limit 1 through llap-ext-client.

Issue Links

• links to: GitHub Pull Request #706