[HIVE-25553] Support Map data-type natively in Arrow format

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.1.2
• Fix Version/s: 4.0.0-alpha-1
• Component/s: llap, Serializers/Deserializers
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: Sruthi Mooriyathvariam Assign to me
• Reporter: Adesh Kumar Rao
• Created: 24/Sep/21 07:07
• Updated: 17/Nov/22 08:48
• Resolved: 27/Oct/21 07:54

Description

Currently ArrowColumnarBatchSerDe converts map datatype as a list of structs data-type (where stuct is containing the key-value pair of the map). This causes issues when reading Map datatype using llap-ext-client as it reads a list of structs instead. 

HiveWarehouseConnector which uses the llap-ext-client throws exception when the schema (containing Map data type) is different from actual data (list of structs).

 

Fixing this issue requires upgrading arrow version (where map data-type is supported), modifying ArrowColumnarBatchSerDe and corresponding Serializer/Deserializer to not use list as a workaround for map and use the arrow map data-type instead. 

Issue Links

• relates to: HIVE-25584 [llap-ext-client] Load data from a Text file for Map dataType is giving errors
• links to: GitHub Pull Request #2689
• links to: GitHub Pull Request #2751

Sub-Tasks

• Upgrade arrow version to 0.15: Closed
• ArrowColumnarBatchSerDe should store map natively instead of converting to list: Closed
• Remove com.vlkan.flatbuffers dependency from serde: Closed