[IMPALA-5790] Failure to get reservation should print buffer pool limit

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 2.10.0
• Component/s: Backend
• Labels: supportability usability
• Target Version: Impala 2.12.0
• Epic Color: ghx-label-6
• Assignee: Unassigned
• Reporter: Matthew Jacobs
• Created: 11/Aug/17 02:04
• Updated: 20/Feb/18 06:43
• Resolved: 20/Feb/18 06:43

Description

When playing around with a custom buffer pool limit, i.e. --buffer_pool_limit, I found that failure to reserve enough memory error message prints the mem tracker hierarchy but does not include the buffer_pool_limit.

E.g.

start-impala-cluster.py --impalad_args="--buffer_pool_limit=2300m "

[localhost:21001] > set DEFAULT_SPILLABLE_BUFFER_SIZE=33554432;
DEFAULT_SPILLABLE_BUFFER_SIZE set to 33554432
[localhost:21001] > set MIN_SPILLABLE_BUFFER_SIZE=33554432;
MIN_SPILLABLE_BUFFER_SIZE set to 33554432
[localhost:21001] > select straight_join * from tpch_parquet.lineitem join (select n_regionkey, count(distinct n_nationkey) from tpch_parquet.nation group by 1) v on n_regionkey = l_orderkey;
Query: select straight_join * from tpch_parquet.lineitem join (select n_regionkey, count(distinct n_nationkey) from tpch_parquet.nation group by 1) v on n_regionkey = l_orderkey
Query submitted at: 2017-08-10 18:54:14 (Coordinator: http://mj-desktop.ca.cloudera.com:25001)
ERROR: ExecQueryFInstances rpc query_id=b94a87085426982d:641c5a9600000000 failed: Failed to get minimum memory reservation of 1.59 GB on daemon mj-desktop.ca.cloudera.com:22000 for query b94a87085426982d:641c5a9600000000 because it would exceed an applicable query, request pool or process memory limit. Memory usage:
Process: Limit=8.35 GB Total=1.11 GB Peak=1.94 GB
  Buffer Pool: Free Buffers: Total=768.00 MB
  Buffer Pool: Clean Pages: Total=0
  Free Disk IO Buffers: Total=256.00 KB Peak=68.00 MB
  RequestPool=default-pool: Total=621.45 MB Peak=1.75 GB
    Query(2540ad2fb916785d:17301f8a00000000): Reservation=544.00 MB ReservationLimit=6.68 GB OtherMemory=77.45 MB Total=621.45 MB Peak=1.75 GB
      Fragment 2540ad2fb916785d:17301f8a00000000: Reservation=0 OtherMemory=16.00 KB Total=16.00 KB Peak=16.00 KB
        EXCHANGE_NODE (id=8): Total=0 Peak=0
        DataStreamRecvr: Total=0 Peak=0
        PLAN_ROOT_SINK: Total=0 Peak=0
        CodeGen: Total=0 Peak=0
      Fragment 2540ad2fb916785d:17301f8a00000005: Reservation=544.00 MB OtherMemory=77.43 MB Total=621.43 MB Peak=621.43 MB
        Runtime Filter Bank: Total=1.00 MB Peak=1.00 MB
        HASH_JOIN_NODE (id=4): Reservation=544.00 MB OtherMemory=42.25 KB Total=544.04 MB Peak=544.06 MB
          Exprs: Total=13.12 KB Peak=13.12 KB
          Hash Join Builder (join_node_id=4): Total=13.12 KB Peak=21.12 KB
            Hash Join Builder (join_node_id=4) Exprs: Total=13.12 KB Peak=13.12 KB
        HDFS_SCAN_NODE (id=0): Total=76.35 MB Peak=76.35 MB
        EXCHANGE_NODE (id=7): Total=0 Peak=0
        DataStreamRecvr: Total=0 Peak=20.04 KB
        DataStreamSender (dst_id=8): Total=1.27 KB Peak=1.27 KB
        CodeGen: Total=21.40 KB Peak=1.41 MB
      Fragment 2540ad2fb916785d:17301f8a00000002: Reservation=0 OtherMemory=0 Total=0 Peak=1.06 GB
        AGGREGATION_NODE (id=3): Reservation=0 OtherMemory=0 Total=0 Peak=544.02 MB
        AGGREGATION_NODE (id=6): Reservation=0 OtherMemory=0 Total=0 Peak=544.04 MB
        EXCHANGE_NODE (id=5): Total=0 Peak=0
        DataStreamRecvr: Total=0 Peak=20.20 KB
        DataStreamSender (dst_id=7): Total=0 Peak=34.90 KB
        CodeGen: Total=0 Peak=529.00 KB
      Fragment 2540ad2fb916785d:17301f8a00000001: Reservation=0 OtherMemory=0 Total=0 Peak=160.21 MB
        AGGREGATION_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=160.02 MB
        HDFS_SCAN_NODE (id=1): Total=0 Peak=157.00 KB
        DataStreamSender (dst_id=5): Total=0 Peak=25.59 KB
        CodeGen: Total=0 Peak=417.50 KB
    Query(b94a87085426982d:641c5a9600000000): Reservation=0 ReservationLimit=6.68 GB OtherMemory=0 Total=0 Peak=0
  Untracked Memory: Total=-266538560.00 B

Issue Links

• duplicates: IMPALA-6222 Make it easier to root-cause "failed to get minimum memory reservation" error
• is related to: IMPALA-6222 Make it easier to root-cause "failed to get minimum memory reservation" error