Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20628

Parsing error when using a complex map data type under dynamic column masking

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: Hive, HiveServer2, Parser, Security
    • Labels:
      None
    • Environment:

      The error can be simulated using HDP 2.6.4 sandbox

      Description

      When trying to use the map complex data type as part of dynamic column mask, Hive throws a parsing error as it is expecting a primitive type (see trace pasted below). The use case is trying to apply masking to elements within a map type by applying a custom hive UDF (to apply the mask) using Ranger. Expect Hive to support complex data types for masking in addition to the primitive types. The expectation occurs when Hive need to evaluate the UDF or apply a standard mask (pass-through works as expected). You can recreate the problem by creating a simple table with a map data type column, then applying the masking to that column through a Ranger resource based policy and  a custom function (you can use a standard Hive UDF  str_to_map('F4','XXXX') to simulate returning a map). 

      CREATE  TABLE `mask_test`(
      `key` string,
      `value` map<string,string>)
      STORED AS INPUTFORMAT
      'org.apache.hadoop.mapred.TextInputFormat'

       

      INSERT INTO TABLE mask_test
      SELECT 'AAA' as key,
      map('F1','2022','F2','2222','F3','3333333') as value
      FROM (select 1 ) as temp;

       

       

      Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.SemanticException:org.apache.hadoop.hive.ql.parse.ParseException: line 1:57 cannot recognize input near 'map' '<' 'string' in primitive type specification
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(SemanticAnalyzer.java:10370)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10486)
      at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:219)
      at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
      at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:465)
      at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:321)
      at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1224)
      at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1218)
      at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:146)
      ... 15 more
      Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.ParseException:line 1:57 cannot recognize input near 'map' '<' 'string' in primitive type specification
      at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:214)
      at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:171)
      at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(SemanticAnalyzer.java:10368)

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              darryl.dutton@t4g.com Darryl Dutton
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: