[IMPALA-8790] IllegalStateException: Illegal reference to non-materialized slot

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.1.0, Impala 3.2.0
• Fix Version/s: Impala 3.3.0
• Component/s: Frontend
• Target Version: Impala 3.3.0
• Epic Color: ghx-label-8
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 24/Jul/19 21:18
• Updated: 15/Oct/20 13:45
• Resolved: 17/Aug/19 12:50

Description

Reproduce:

$ hdfs dfs -put foo.parq hdfs:///tmp
impala> create table foo (uid string, cid string) stored as parquet;
impala> load data inpath 'hdfs:///tmp/foo.parq' into table foo;

With the stats, the following query hits an IllegalStateException:

impala> compute stats foo;
impala> explain select uid, cid,
   rank() over (partition by uid order by count(*) desc)
from (select uid, cid from foo) w
group by uid, cid;

ERROR: IllegalStateException: Illegal reference to non-materialized slot: tid=1 sid=2

Get the stacktrace in impalad.INFO:

I1015 13:44:25.276245 67623 jni-util.cc:256] 474111eba829d732:233f6caa00000000] java.lang.IllegalStateException: Illegal reference to non-materialized slot: tid=1 sid=2
        at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
        at org.apache.impala.analysis.SlotRef.toThrift(SlotRef.java:161)
        at org.apache.impala.analysis.Expr.treeToThriftHelper(Expr.java:752)
        at org.apache.impala.analysis.Expr.treeToThrift(Expr.java:731)
        at org.apache.impala.analysis.Expr.treesToThrift(Expr.java:783)
        at org.apache.impala.planner.DataPartition.toThrift(DataPartition.java:89)
        at org.apache.impala.planner.PlanFragment.toThrift(PlanFragment.java:338)
        at org.apache.impala.service.Frontend.createPlanExecInfo(Frontend.java:1146)
        at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1185)
        at org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1466)
        at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1345)
        at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1252)
        at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1222)
        at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:167)
I1015 13:44:25.277076 67623 status.cc:124] 474111eba829d732:233f6caa00000000] IllegalStateException: Illegal reference to non-materialized slot: tid=1 sid=2 

Without the stats, it runs successfully:

impala> drop stats foo;
impala> explain select uid, cid,
   rank() over (partition by uid order by count(*) desc)
from (select uid, cid from foo) w
group by uid, cid;
+------------------------------------------------------------------------------------+
| Explain String                                                                     |
+------------------------------------------------------------------------------------+
| Max Per-Host Resource Reservation: Memory=84.02MB Threads=5                        |
| Per-Host Resource Estimates: Memory=304MB                                          |
| WARNING: The following tables are missing relevant table and/or column statistics. |
| common_action.foo                                                                  |
|                                                                                    |
| PLAN-ROOT SINK                                                                     |
| |                                                                                  |
| 07:EXCHANGE [UNPARTITIONED]                                                        |
| |                                                                                  |
| 03:ANALYTIC                                                                        |
| |  functions: rank()                                                               |
| |  partition by: uid                                                               |
| |  order by: count(*) DESC                                                         |
| |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW                       |
| |  row-size=40B cardinality=1.10K                                                  |
| |                                                                                  |
| 02:SORT                                                                            |
| |  order by: uid ASC NULLS FIRST, count(*) DESC                                    |
| |  row-size=32B cardinality=1.10K                                                  |
| |                                                                                  |
| 06:EXCHANGE [HASH(uid)]                                                            |
| |                                                                                  |
| 05:AGGREGATE [FINALIZE]                                                            |
| |  output: count:merge(*)                                                          |
| |  group by: uid, cid                                                              |
| |  row-size=32B cardinality=1.10K                                                  |
| |                                                                                  |
| 04:EXCHANGE [HASH(uid,cid)]                                                        |
| |                                                                                  |
| 01:AGGREGATE [STREAMING]                                                           |
| |  output: count(*)                                                                |
| |  group by: uid, cid                                                              |
| |  row-size=32B cardinality=1.10K                                                  |
| |                                                                                  |
| 00:SCAN HDFS [common_action.foo]                                                   |
|    HDFS partitions=1/1 files=1 size=5.19KB                                         |
|    row-size=24B cardinality=1.10K                                                  |
+------------------------------------------------------------------------------------+
Fetched 37 row(s) in 0.03s

 

Issue Links

• is broken by: IMPALA-110 Add support for multiple distinct operators in the same query block
• links to: Code Review