Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
3.1.0, 3.1.2
Description
Currently ArrowColumnarBatchSerDe converts map datatype as a list of structs data-type (where stuct is containing the key-value pair of the map). This causes issues when reading Map datatype using llap-ext-client as it reads a list of structs instead.
HiveWarehouseConnector which uses the llap-ext-client throws exception when the schema (containing Map data type) is different from actual data (list of structs).
Fixing this issue requires upgrading arrow version (where map data-type is supported), modifying ArrowColumnarBatchSerDe and corresponding Serializer/Deserializer to not use list as a workaround for map and use the arrow map data-type instead.
Attachments
Issue Links
- relates to
-
HIVE-25584 [llap-ext-client] Load data from a Text file for Map dataType is giving errors
- Closed
- links to
pvary kgyrtkirk ShubhamChaurasia These changes are backward incompatible (not using list to store map).
But since this is being used internally by llap (and creating hive tables with arrow format is not supported?), it should not cause any issues.
Let me know if you have any concerns.