[IMPALA-1619] Support single allocations larger 1GB

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.0.1
• Fix Version/s: Impala 2.7.0, Impala 2.6.3
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 2.7.0, Impala 2.6.3
• Assignee: Michael Ho
• Reporter: Martin Grund
• Created: 17/Dec/14 23:13
• Updated: 12/Oct/16 05:39
• Resolved: 11/Jul/16 21:14

Description

Currently, most of the allocations are managed via the FreePool class. Unfortunately, this class has a sever drawback that it uses a signed 32 bit integer to specify allocations. Interestingly, the FreePool manages a list of 64 sub-lists with free blocks, but the upper half of these lists is never used. The FreePool uses a MemPool to perform the actual allocation, but the MemPool only uses a 32 bit signed integer as well to indicate the allocation size.

If an allocation requests more than 1GB of data, the 32 bit integer will overflow and request a negative amount of memory which will ultimately lead to a crash.

To reproduce the crash issue:

select length(group_concat(l_comment, "!")) from (select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem union all
select l_comment from tpch_parquet.lineitem) a;

Issue Links

• is blocked by: IMPALA-2284 Handle large string allocations (>1GB) in built-in UDFs gracefully
• is duplicated by: IMPALA-2814 Impala daemons crash with large Snappy compressed text files
• is related to: IMPALA-2065 Crash: impala::BufferedBlockMgr::ConsumeMemory (PartitionedHashJoinNode)
• is related to: IMPALA-3069 mem-pool.cc:254] Check failed: chunks_[i].allocated_bytes > 0 (-2104452104 vs. 0)
• relates to: IMPALA-2249 Avoid allocating stringbuffer larger than 1GB in HdfsTextScanner::FillByteBufferCompressedFile()