[IMPALA-6996] PartitionedAggregationNode::Close() should not dump stack trace

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 2.12.0
• Component/s: Backend
• Epic Color: ghx-label-9
• Assignee: Unassigned
• Reporter: Zoram Thanga
• Created: 08/May/18 21:35
• Updated: 11/May/18 05:24
• Resolved: 11/May/18 05:24

Description

A CTAS query with MEM_LIMIT=6GB hit the limit while in an AggregationNode. Upon processing the failure, the fragment instance that hit the MEM_LIMIT tries to allocate more memory, and repeatedly dumps the following error message and stack trace to Impalad log file, once per row.

I0508 01:32:10.611563 40614 status.cc:55] Memory limit exceeded: FunctionContextImpl::AllocateLocal's allocations exceeded memory limits.
Exprs could not allocate 3.00 B without exceeding limit.
Error occurred on backend <redacted>.com:22000 by fragment b429287bdabe184:9e7b5e960000002c
Memory left in process limit: 384.47 GB
Memory left in query limit: -6947009.00 B
Query(b429287bdabe184:9e7b5e9600000000): memory limit exceeded. Limit=6.00 GB Total=6.01 GB Peak=6.01 GB
  Fragment b429287bdabe184:9e7b5e960000002c: Total=2.76 GB Peak=2.76 GB
    AGGREGATION_NODE (id=3): Total=2.75 GB Peak=2.75 GB
      Exprs: Total=751.94 MB Peak=751.94 MB
    EXCHANGE_NODE (id=2): Total=0 Peak=0
    DataStreamRecvr: Total=7.42 MB Peak=7.42 MB
    CodeGen: Total=5.57 KB Peak=745.50 KB
  Block Manager: Limit=4.80 GB Total=4.39 GB Peak=4.40 GB
  Fragment b429287bdabe184:9e7b5e960000000c: Total=3.25 GB Peak=3.25 GB
    AGGREGATION_NODE (id=1): Total=3.19 GB Peak=3.19 GB
      Exprs: Total=816.06 MB Peak=816.06 MB
    HDFS_SCAN_NODE (id=0): Total=59.96 MB Peak=200.09 MB
    CodeGen: Total=5.96 KB Peak=876.00 KB
    @           0x83c78a  impala::Status::Status()
    @           0x83c98e  impala::Status::MemLimitExceeded()
    @           0xa24344  impala::MemTracker::MemLimitExceeded()
    @           0xa35ccd  impala::RuntimeState::SetMemLimitExceeded()
    @           0xb6bd6d  impala::FunctionContextImpl::CheckAllocResult()
    @           0xb6ae78  impala::FunctionContextImpl::AllocateLocal()
    @           0xb6b10f  impala_udf::StringVal::StringVal()
    @           0xb6b16a  impala_udf::StringVal::CopyFrom()
    @           0x8a2641  impala::AggregateFunctions::StringValGetValue()
    @           0x8a2661  impala::AggregateFunctions::StringValSerializeOrFinalize()
    @           0xd411c5  impala::AggFnEvaluator::SerializeOrFinalize()
    @           0xce6569  impala::PartitionedAggregationNode::CleanupHashTbl()
    @           0xce693b  impala::PartitionedAggregationNode::Partition::Close()
    @           0xce8152  impala::PartitionedAggregationNode::ClosePartitions()
    @           0xcf163e  impala::PartitionedAggregationNode::Close()
    @           0xa7ae35  impala::FragmentInstanceState::Close()
    @           0xa7ebfb  impala::FragmentInstanceState::Exec()
    @           0xa6aaf6  impala::QueryState::ExecFInstance()
    @           0xbef9d9  impala::Thread::SuperviseThread()
    @           0xbf0394  boost::detail::thread_data<>::run()
    @           0xe588aa  (unknown)
    @     0x7f8a54c32e25  start_thread
    @     0x7f8a5496034d  __clone

Here't the profile summary:

Operator       #Hosts  Avg Time  Max Time    #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail                       
-----------------------------------------------------------------------------------------------------------------------
03:AGGREGATE       32   0.000ns   0.000ns        0       1.06B    2.75 GB      165.23 GB  FINALIZE                     
02:EXCHANGE        32   1s246ms   7s429ms  412.85M       1.06B          0              0  HASH(<redacted>) 
01:AGGREGATE       32  22s539ms  27s631ms  422.16M       1.06B    3.09 GB      165.23 GB  STREAMING                    
00:SCAN HDFS       32   1s221ms   2s093ms  810.12M       1.06B  222.22 MB      616.00 MB  <redacted>      
    Errors: Memory limit exceeded: Error occurred on backend <redacted>.com:22000 by fragment b429287bdabe184:9e7b5e960000002c
Memory left in process limit: 386.76 GB
Memory left in query limit: -19080.00 B

And the query string:

create TABLE TT><redacted>
        STORED AS PARQUET
        TBLPROPERTIES('parquet.compression'='SNAPPY')
T>AS 
T>SELECT  T>        <redacted>,
                                <redacted>, 
                                <redacted>,
                                MAX(<redacted>) AS <redacted>,
                                MAX(<redacted>) AS <redacted>,
                                MAX(<redacted>) AS <redacted>,
                                MAX(<redacted>) AS  <redacted>               
T>FROM TT>        <redacted>
        GROUP BY                <redacted>,
                                <redacted>,
                                <redacted>
    Coordinator: <redacted>.com:22000
    Query Options (non default): MEM_LIMIT=6442450944,REQUEST_POOL=<redacted>,SYNC_DDL=1,MT_DOP=0
    DDL Type: CREATE_TABLE_AS_SELECT

Issue Links

• is duplicated by: IMPALA-6997 Query execution should notice UDF MemLimitExceeded errors more quickly
• relates to: IMPALA-6997 Query execution should notice UDF MemLimitExceeded errors more quickly