[IMPALA-2284] Handle large string allocations (>1GB) in built-in UDFs gracefully

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.0.1
• Fix Version/s: Impala 2.3.0, Impala 2.2.8
• Labels: crash resource-management
• Target Version: Impala 2.3.0, Impala 2.2.8
• Assignee: Martin Grund
• Reporter: Martin Grund
• Created: 02/Sep/15 05:33
• Updated: 29/Jan/16 20:26
• Resolved: 23/Sep/15 22:16

Description

When group_concat() is used in large queries, it can happen that the UDF wants to allocate more than 1GB of data. If this is the case, Impala will crash.

To reproduce:

select length(group_concat(l_comment, "!")) from (select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem) a;

callstack

#0 0x0000003eaf232925 in raise () from /lib64/libc.so.6 
#1 0x0000003eaf234105 in abort () from /lib64/libc.so.6 
#2 0x00007f4eba0a9155 in os::abort(bool) () from /usr/java/jdk1.7.0_45-cloudera/jre/lib/amd64/server/libjvm.so 
#3 0x00007f4eba228087 in VMError::report_and_die() () from /usr/java/jdk1.7.0_45-cloudera/jre/lib/amd64/server/libjvm.so 
#4 0x00007f4eba0adadf in JVM_handle_linux_signal () from /usr/java/jdk1.7.0_45-cloudera/jre/lib/amd64/server/libjvm.so 
#5 <signal handler called> 
#6 0x00000000009965ba in impala_udf::FunctionContext::Allocate(int) () 
#7 0x000000000078f3c8 in impala::AggregateFunctions::StringConcat(impala_udf::FunctionContext*, impala_udf::StringVal const&, impala_udf::StringVal const&, impala_udf::StringVal*) () 

Issue Links

• blocks: IMPALA-1619 Support single allocations larger 1GB