[IMPALA-13302] Some ExprRewriteRule results are not analyzed, leading to unmaterialized slots from reAnalyze

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.3.0
• Fix Version/s: Impala 4.5.0
• Component/s: Frontend
• Target Version: Impala 4.4.2
• Epic Color: ghx-label-10
• Assignee: Michael Smith
• Reporter: Michael Smith
• Created: 15/Aug/24 19:08
• Updated: 24/Sep/24 22:44
• Resolved: 24/Sep/24 22:43

Description

IMPALA-12164 skipped registering conjuncts that the analyzer expects to remove because an earlier conjunct evaluates to constant False. However some ExprRewriteRules don't analyze the predicates they produce, which can lead to those conjuncts not actually being removed until a reAnalyze phase.

reAnalyze uses a new Analyzer (with new GlobalState); it restarts counting Expr IDs from 0. That can lead to re-using the same Expr ID and marking it as assigned. Then when a new Expr gets the same ID, it will skip materializing slots, which can cause problems later (like if that Expr is part of a hash join).

Some example queries:
1. still a problem

WITH v AS (SELECT 1 FROM functional.alltypestiny t1
  JOIN functional.alltypestiny t2 ON t1.id = t2.id)
SELECT 1
FROM functional.alltypestiny t1
WHERE ((t1.id = 1 and false) or (t1.id = 1 and false))
  AND t1.id = 1 AND t1.id = 1
UNION ALL
SELECT 1
FROM functional.alltypestiny t1
WHERE ((t1.id = 1 and false) or (t1.id = 1 and false))
  AND t1.id = 1 AND t1.id = 1
UNION ALL SELECT 1 FROM v
UNION ALL SELECT 1 FROM v;

2. already fixed via IMPALA-13203

WITH v as (SELECT 1 FROM functional.alltypes t1
  JOIN functional.alltypes t2 ON t1.id = t2.id)
SELECT 1 FROM functional.alltypes t1
  WHERE t1.id = 1 AND t1.id = 1 AND t1.id = 1 AND false
UNION ALL
SELECT 1 FROM functional.alltypes t1
  WHERE t1.id = 1 AND false
UNION ALL SELECT 1 FROM v
UNION ALL SELECT 1 FROM v;

Issue Links

• is caused by: IMPALA-12164 The query fails with "IllegalStateException: Illegal reference to non-materialized slot: tid=x sid=x"
• is related to: IMPALA-13360 Set type of assignedConjuncts should be consistent
• is related to: IMPALA-8030 Remove duplicate in-clause values for selectivity calcs
• relates to: IMPALA-13203 ExprRewriter did not rewrite 'id = 0 OR false' as expected
• relates to: IMPALA-13344 Impala rewrites may be incomplete
• Testing discovered: IMPALA-13365 markConjunctAssigned called on conjuncts that aren't registered