[IMPALA-10865] Query with select alias the same as column name rewrite error for grouping exprs

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0
• Fix Version/s: Impala 4.2.0, Impala 4.1.1
• Epic Color: ghx-label-8
• Assignee: guojingfeng
• Reporter: guojingfeng
• Created: 17/Aug/21 03:50
• Updated: 30/Aug/22 12:10
• Resolved: 28/Jul/22 07:41

Description

We hit a AnalyzeException when replay online query stmts when upgrading from 3.4.x from 4.0.
The exception is that:

Java exception follows:
org.apache.impala.common.AnalysisException: select list expression not produced by aggregation output (missing from GROUP BY clause?): column column_alias
        at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.verifyAggregation(SelectStmt.java:906)
        at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.analyze(SelectStmt.java:297)
        at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.access$100(SelectStmt.java:265)
        at org.apache.impala.analysis.SelectStmt.analyze(SelectStmt.java:258)
        at org.apache.impala.analysis.AnalysisContext.reAnalyze(AnalysisContext.java:575)
        at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:549)
        at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:445)
        at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1659)
        at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1625)
        at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1595)
        at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:162)
I0802 22:05:34.565348  6026 BaseAuthorizationChecker.java:112] 774fe9cee424b3f0:86fb11dc00000000] Authorization check took 107 ms
I0802 22:05:34.565541  6026 jni-util.cc:286] 774fe9cee424b3f0:86fb11dc00000000] org.apache.impala.common.AnalysisException: select list expression not produced by aggregation output (missing from GROUP BY clause?): test_id test_group
        at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.verifyAggregation(SelectStmt.java:906)
        at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.analyze(SelectStmt.java:297)
        at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.access$100(SelectStmt.java:265)
        at org.apache.impala.analysis.SelectStmt.analyze(SelectStmt.java:258)
        at org.apache.impala.analysis.AnalysisContext.reAnalyze(AnalysisContext.java:575)
        at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:549)
        at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:445)
        at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1659)
        at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1625)
        at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1595)
        at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:162)
I0802 22:05:34.565551  6026 status.cc:129] 774fe9cee424b3f0:86fb11dc00000000] AnalysisException: select list expression not produced by aggregation output (missing from GROUP BY clause?): column column_alias
    @           0xd6c6d3
    @          0x159e8e9
    @          0x133545b
    @          0x19e704b
    @          0x13a46b1
    @          0x13b67d2
    @          0x14a3089
    @          0x14574d7
    @          0x13c2177
    @           0xcc9af5
    @          0x1128176
    @          0x111c058
    @          0x16649e1
    @          0x1665e5a
    @          0x1ea21c1
    @     0x7f20726f8e24
    @     0x7f206f20b35c

This is a corner case of SelectStmt analyze.
In general, the analyze algorithm is that:
1. Analyze the stmt tree and make copies of expressions
2. Rewrite selected expressions
3. Make copied expressions analyzed
4. ReAnalyze the tree // This is the root cause

The problem maybe is that step 1~3 changed the state of expr tree but not keep in sync with re-analyze phase.

 

Reproduce case:

EXPLAIN
SELECT ss_item_sk ss_item_sk_group,
       ss_item_sk+300 ss_item_sk,
       count(ss_ticket_number)
FROM tpcds.store_sales a
WHERE ss_sold_date_sk > cast('245263' AS INT)
GROUP BY ss_item_sk_group,
         ss_item_sk;

 

Issue Links

• relates to: IMPALA-10096 May throw exception after expr rewritten , if the group by ordinal reference is still a numeric literal
• relates to: IMPALA-9620 Predicates in the SELECT and GROUP-BY cause failure with CNF rewrite enabled
• relates to: IMPALA-9693 Predicate in the ORDER BY clause causes failure with cnf rewrite enabled
• links to: Code Review