[IMPALA-2284] Handle large string allocations (>1GB) in built-in UDFs gracefully - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: Impala 2.0.1
Fix Version/s: Impala 2.3.0, Impala 2.2.8
Component/s: None
Labels:
- crash
- resource-management

Target Version:

Impala 2.3.0, Impala 2.2.8

Description

When group_concat() is used in large queries, it can happen that the UDF wants to allocate more than 1GB of data. If this is the case, Impala will crash.

To reproduce:

select length(group_concat(l_comment, "!")) from (select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem) a;

callstack

#0 0x0000003eaf232925 in raise () from /lib64/libc.so.6 
#1 0x0000003eaf234105 in abort () from /lib64/libc.so.6 
#2 0x00007f4eba0a9155 in os::abort(bool) () from /usr/java/jdk1.7.0_45-cloudera/jre/lib/amd64/server/libjvm.so 
#3 0x00007f4eba228087 in VMError::report_and_die() () from /usr/java/jdk1.7.0_45-cloudera/jre/lib/amd64/server/libjvm.so 
#4 0x00007f4eba0adadf in JVM_handle_linux_signal () from /usr/java/jdk1.7.0_45-cloudera/jre/lib/amd64/server/libjvm.so 
#5 <signal handler called> 
#6 0x00000000009965ba in impala_udf::FunctionContext::Allocate(int) () 
#7 0x000000000078f3c8 in impala::AggregateFunctions::StringConcat(impala_udf::FunctionContext*, impala_udf::StringVal const&, impala_udf::StringVal const&, impala_udf::StringVal*) ()