[IMPALA-10838] Error when struct returned from WITH() and used in an ORDER BY

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Frontend
• Labels: complextype nested_types
• Epic Link: Fix complex type related issues
• Target Version: Impala 4.2.0
• Epic Color: ghx-label-6
• Assignee: Daniel Becker
• Reporter: Gabor Kaszab
• Created: 04/Aug/21 11:11
• Updated: 31/Oct/22 04:48
• Resolved: 02/May/22 07:35

Description

with sub as (
    select id, small_struct
    from functional_orc_def.complextypes_structs
    where length(small_struct.s) > 5)
select sub.id, sub.small_struct from sub order by sub.small_struct.i desc;

The above query results an error when trying to run SlotRef.toThrift()

ERROR: IllegalStateException: Illegal reference to non-materialized tuple: debugname=InlineViewRef sub alias=sub tid=2

If I rewrite the query a bit to return the member of the struct from the inline view (WITH()) and use this in the ORDER by then the query succeeds as expected:

with sub as (
    select id, small_struct, small_struct.i as si
    from functional_orc_def.complextypes_structs where small_struct.i > 19200)
select sub.id, sub.small_struct from sub order by sub.si desc;

In SortNode.toThrift() I checked what the sort exprs and the resolved tuple exprs are and I see a difference that could be the cause.
In the problematic case:

- sort exprs in SortNode:
  SlotRef{label=small_struct.i, type=INT, id=15}
- resolved exprs in SortNode: 
  SlotRef{label=id, path=id, type=INT, id=0} 
  SlotRef{label=small_struct, path=small_struct, type=STRUCT<i:INT,s:STRING>, id=1} 
  *SlotRef{label=sub.small_struct.i, path=sub.small_struct.i, type=INT, id=10}*

In the successful case:

- sort exprs in SortNode: 
  SlotRef{label=si, type=INT, id=14}
- resolved exprs in SortNode: 
  SlotRef{label=id, path=id, type=INT, id=0} 
  SlotRef{label=small_struct, path=small_struct, type=STRUCT<i:INT,s:STRING>, id=1} 
  *SlotRef{label=small_struct.i, path=small_struct.i, type=INT, id=4}*

Issue Links

• depends upon: IMPALA-9495 Allow Struct type in SELECT list for ORC tables
• fixes: IMPALA-11067 Unify struct subexpressions in rows
• relates to: IMPALA-11685 Slot memory sharing between struct and field not working if the field is also a struct
• relates to: IMPALA-11687 Select * with EXPAND_COMPLEX_TYPES=1 and explicit complex types fails