[HIVE-20628] Parsing error when using a complex map data type under dynamic column masking

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.1.0
• Component/s: Hive, HiveServer2, Parser, Security
• Labels: pull-request-available

Environment

The error can be simulated using HDP 2.6.4 sandbox

• Assignee: Unassigned
• Reporter: Darryl Dutton
• Created: 24/Sep/18 14:15
• Updated: 07/Jul/22 15:24
• Resolved: 07/Jul/22 15:24

Description

When trying to use the map complex data type as part of dynamic column mask, Hive throws a parsing error as it is expecting a primitive type (see trace pasted below). The use case is trying to apply masking to elements within a map type by applying a custom hive UDF (to apply the mask) using Ranger. Expect Hive to support complex data types for masking in addition to the primitive types. The expectation occurs when Hive need to evaluate the UDF or apply a standard mask (pass-through works as expected). You can recreate the problem by creating a simple table with a map data type column, then applying the masking to that column through a Ranger resource based policy and  a custom function (you can use a standard Hive UDF  str_to_map('F4','XXXX') to simulate returning a map). 

CREATE  TABLE `mask_test`(
`key` string,
`value` map<string,string>)
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'

 

INSERT INTO TABLE mask_test
SELECT 'AAA' as key,
map('F1','2022','F2','2222','F3','3333333') as value
FROM (select 1 ) as temp;

 

 

Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.SemanticException:org.apache.hadoop.hive.ql.parse.ParseException: line 1:57 cannot recognize input near 'map' '<' 'string' in primitive type specification
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(SemanticAnalyzer.java:10370)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10486)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:219)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:465)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:321)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1224)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1218)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:146)
... 15 more
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.ParseException:line 1:57 cannot recognize input near 'map' '<' 'string' in primitive type specification
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:214)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:171)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(SemanticAnalyzer.java:10368)

Issue Links

• is duplicated by: HIVE-22822 Column masking policies on complex column will cause unrelative query failures
• is related to: HIVE-26377 Support complex types for masking
• relates to: HIVE-26378 Improve error message for masking over complex data types
• links to: GitHub Pull Request #3417