[IMPALA-8718] Incorrect AnalysisException with outer join complex type column

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.10.0
• Fix Version/s: Impala 3.4.0
• Component/s: Frontend
• Labels: planner
• Epic Color: ghx-label-3
• Assignee: Quanlong Huang
• Reporter: Tamas Mate
• Created: 27/Jun/19 13:25
• Updated: 14/May/20 03:26
• Resolved: 11/Sep/19 23:45

Description

Although user is not explicitly specifying IS NOT NULL predicate the query fails with:

org.apache.impala.common.AnalysisException: IS NOT NULL predicate does not support complex types: col3 IS NOT NULL.

When a complex type is on the right hand side of the join it is wrapped by IsNullPredicate, as it could be null at the end of the join. Which is caught by this condition later. The following exception is thrown:

I0620 04:11:29.498865 474227 jni-util.cc:211] java.lang.IllegalStateException: org.apache.impala.common.AnalysisException: IS NOT NULL predicate does not support complex types: col3 IS NOT NULL
  at org.apache.impala.analysis.Expr.analyzeNoThrow(Expr.java:362)
  at org.apache.impala.analysis.TupleIsNullPredicate.requiresNullWrapping(TupleIsNullPredicate.java:158)
  at org.apache.impala.analysis.TupleIsNullPredicate.wrapExpr(TupleIsNullPredicate.java:133)
  at org.apache.impala.analysis.TupleIsNullPredicate.wrapExprs(TupleIsNullPredicate.java:122)
  at org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(SingleNodePlanner.java:1042)
  at org.apache.impala.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:1454)
  at org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(SingleNodePlanner.java:778)
  at org.apache.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:616)
  at org.apache.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:259)
  at org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:149)
  at org.apache.impala.planner.Planner.createPlan(Planner.java:98)
  at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1005)
  at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1101)
  at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156)
Caused by: org.apache.impala.common.AnalysisException: IS NOT NULL predicate does not support complex types: col3 IS NOT NULL
  at org.apache.impala.analysis.IsNullPredicate.analyzeImpl(IsNullPredicate.java:127)
  at org.apache.impala.analysis.Expr.analyze(Expr.java:343)
  at org.apache.impala.analysis.Expr.analyzeNoThrow(Expr.java:360)
  ... 13 more

I believe the nullable is necessary, but the error message for this condition is incorrect. The issue can be reproduced with the following queries.

create table sample_test_1 
(col1 string,
col2 string,
col3 array<struct<col31:string,col32:int>>);

create table sample_test_2 
(col1 string,
col2 string);

with leftSide as
(
select col1
  from sample_test_2
),
rightSide as
(
select t.col1,
       rank() over(order by t.col1) as rnk
  from sample_test_1 t
      left outer join t.col3
)
select *
  from leftSide l
       left join rightSide r
               on l.col1 = r.col1

cc.: ychena