[IMPALA-10387] Implement missing overloads of mask functions used in Ranger default masking policies

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.4.0
• Fix Version/s: Impala 4.0.0
• Component/s: Frontend
• Epic Link: Ranger Column Masking
• Epic Color: ghx-label-12
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 10/Dec/20 07:48
• Updated: 15/Jan/21 23:36
• Resolved: 15/Jan/21 23:36

Description

Some meaningless overloads of the mask functions are not implemented in Impala since Hive always returns NULL on them. It's expected to use the mask_null function (Nullify policy) directly.

However, it's common for users to configure Redact policy (mask) on all columns. If there are float/double/decimal columns, they will hit errors like this in Impala:

I1208 22:38:03.880369  8354 jni-util.cc:288] b8461c997916681b:83957ca300000000] org.apache.impala.common.AnalysisException: No matching function with signature: mask(FLOAT).
        at org.apache.impala.analysis.FunctionCallExpr.analyzeImpl(FunctionCallExpr.java:555)
        at org.apache.impala.analysis.Expr.analyze(Expr.java:438)
        at org.apache.impala.analysis.Expr.analyze(Expr.java:432)
        at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.analyzeSelectClause(SelectStmt.java:276)
        at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.analyze(SelectStmt.java:215)
        at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.access$100(SelectStmt.java:199)
        at org.apache.impala.analysis.SelectStmt.analyze(SelectStmt.java:192)
        at org.apache.impala.analysis.InlineViewRef.analyze(InlineViewRef.java:218)
        at org.apache.impala.analysis.FromClause.analyze(FromClause.java:78)
        at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.analyze(SelectStmt.java:213)
        at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.access$100(SelectStmt.java:199)
        at org.apache.impala.analysis.SelectStmt.analyze(SelectStmt.java:192)
        at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:473)
        at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:437)
        at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1530)
        at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1497)
        at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1467)
        at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:154) 

We'd better add these overloads to avoid hitting such errors in simple setup. These patterns are used by Ranger default masking policies:

• mask(col)
• mask_show_last_n(col, 4, 'x', 'x', 'x', -1, '1')
• mask_show_first_n(col, 4, 'x', 'x', 'x', -1, '1')
• mask_hash(col)
• mask(col, 'x', 'x', 'x', -1, '1', 1, 0, -1)

We should have overloads for col to be any types.

Issue Links

• relates to: IMPALA-9010 Support pre-defined mask types from Ranger UI