[HIVE-17406] UDAF throws IllegalArgumentException for a complex input when column stats is not provided

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 2.3.0
• Component/s: Query Processor
• Assignee: A
• Reporter: Makoto Yui
• Created: 29/Aug/17 07:27
• Updated: 22/Dec/21 12:11
• Resolved: 04/Sep/17 09:41

Description

I found that UDAF (both generic and non-generic UDAF w/ or w/o estimable) of Hive v2.3.0 throws IllegalArgumentException for a complex input when column stats is not provided.

The exception does not occur in v2.1.0.

https://github.com/apache/hive/blob/34eebff194e81180202d198200e84058c4910d95/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L1156

select version();
> 2.3.0-amzn-0 rcb482944667f96f43c89932dcb66d61ee7e4ac1d

with t2 as ( 
  select array(1,2) as c1 
  union all 
  select array(2,3) as c1
) 
select collect_list(c1) from t2;

> FAILED: IllegalArgumentException Size requested for unknown type: java.util.Collection

On the other hand, it succeeds when colunm stats is provided as follows:

create table t1 as (
  select array(1,2) as c1 
  union all
  select array(2,3) as c1
);

> select collect_list(c1) from t1;
[[1,2],[2,3]]

> desc formatted t1;
...       
Table Parameters:                
        COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
        numFiles                2                   
        numRows                 2                   
        rawDataSize             6                   
        totalSize               8                   
        transient_lastDdlTime   1503990290

Issue Links

• is fixed by: HIVE-16811 Estimate statistics in absence of stats
• is related to: HIVE-11160 Auto-gather column stats