[SPARK-48027] InjectRuntimeFilter for multi-level join should check child join type - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 4.0.0
Fix Version/s: 4.0.0
Component/s: SQL
Labels:
- pull-request-available

Description

with 
refund_info as (
    select
        loan_id,
        1 as refund_type
    from
        default.table_b
    where grass_date = '2024-04-25'
       
),
next_month_time as (
    select /*+ broadcast(b, c) */
         loan_id
        ,1 as final_repayment_time
    FROM default.table_c
    where grass_date = '2024-04-25'
)
select
        a.loan_id
        ,c.final_repayment_time
        ,b.refund_type    from
        (select
             loan_id
        from
            default.table_a2
        where grass_date = '2024-04-25'
        select
            loan_id
        from
            default.table_a1
        where grass_date = '2024-04-24' 
        ) a
    left join
        refund_info b
    on a.loan_id = b.loan_id
    left join
        next_month_time c
    on a.loan_id = c.loan_id
;

In this query, it inject table_b as table_c's runtime filter, but table_b join condition is LEFT OUTER, causing table_c missing data.

Caused by

InjectRuntimeFilter.extractSelectiveFilterOverScan(), when handle join, since left plan is a UNION< result is NONE, then zip l/r keys to extract from right. Then cause this issue