Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5790

Failure to get reservation should print buffer pool limit

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: Impala 2.10.0
    • Fix Version/s: None
    • Component/s: Backend

      Description

      When playing around with a custom buffer pool limit, i.e. --buffer_pool_limit, I found that failure to reserve enough memory error message prints the mem tracker hierarchy but does not include the buffer_pool_limit.

      E.g.

      start-impala-cluster.py --impalad_args="--buffer_pool_limit=2300m "
      
      [localhost:21001] > set DEFAULT_SPILLABLE_BUFFER_SIZE=33554432;
      DEFAULT_SPILLABLE_BUFFER_SIZE set to 33554432
      [localhost:21001] > set MIN_SPILLABLE_BUFFER_SIZE=33554432;
      MIN_SPILLABLE_BUFFER_SIZE set to 33554432
      [localhost:21001] > select straight_join * from tpch_parquet.lineitem join (select n_regionkey, count(distinct n_nationkey) from tpch_parquet.nation group by 1) v on n_regionkey = l_orderkey;
      Query: select straight_join * from tpch_parquet.lineitem join (select n_regionkey, count(distinct n_nationkey) from tpch_parquet.nation group by 1) v on n_regionkey = l_orderkey
      Query submitted at: 2017-08-10 18:54:14 (Coordinator: http://mj-desktop.ca.cloudera.com:25001)
      ERROR: ExecQueryFInstances rpc query_id=b94a87085426982d:641c5a9600000000 failed: Failed to get minimum memory reservation of 1.59 GB on daemon mj-desktop.ca.cloudera.com:22000 for query b94a87085426982d:641c5a9600000000 because it would exceed an applicable query, request pool or process memory limit. Memory usage:
      Process: Limit=8.35 GB Total=1.11 GB Peak=1.94 GB
        Buffer Pool: Free Buffers: Total=768.00 MB
        Buffer Pool: Clean Pages: Total=0
        Free Disk IO Buffers: Total=256.00 KB Peak=68.00 MB
        RequestPool=default-pool: Total=621.45 MB Peak=1.75 GB
          Query(2540ad2fb916785d:17301f8a00000000): Reservation=544.00 MB ReservationLimit=6.68 GB OtherMemory=77.45 MB Total=621.45 MB Peak=1.75 GB
            Fragment 2540ad2fb916785d:17301f8a00000000: Reservation=0 OtherMemory=16.00 KB Total=16.00 KB Peak=16.00 KB
              EXCHANGE_NODE (id=8): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=0
              PLAN_ROOT_SINK: Total=0 Peak=0
              CodeGen: Total=0 Peak=0
            Fragment 2540ad2fb916785d:17301f8a00000005: Reservation=544.00 MB OtherMemory=77.43 MB Total=621.43 MB Peak=621.43 MB
              Runtime Filter Bank: Total=1.00 MB Peak=1.00 MB
              HASH_JOIN_NODE (id=4): Reservation=544.00 MB OtherMemory=42.25 KB Total=544.04 MB Peak=544.06 MB
                Exprs: Total=13.12 KB Peak=13.12 KB
                Hash Join Builder (join_node_id=4): Total=13.12 KB Peak=21.12 KB
                  Hash Join Builder (join_node_id=4) Exprs: Total=13.12 KB Peak=13.12 KB
              HDFS_SCAN_NODE (id=0): Total=76.35 MB Peak=76.35 MB
              EXCHANGE_NODE (id=7): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=20.04 KB
              DataStreamSender (dst_id=8): Total=1.27 KB Peak=1.27 KB
              CodeGen: Total=21.40 KB Peak=1.41 MB
            Fragment 2540ad2fb916785d:17301f8a00000002: Reservation=0 OtherMemory=0 Total=0 Peak=1.06 GB
              AGGREGATION_NODE (id=3): Reservation=0 OtherMemory=0 Total=0 Peak=544.02 MB
              AGGREGATION_NODE (id=6): Reservation=0 OtherMemory=0 Total=0 Peak=544.04 MB
              EXCHANGE_NODE (id=5): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=20.20 KB
              DataStreamSender (dst_id=7): Total=0 Peak=34.90 KB
              CodeGen: Total=0 Peak=529.00 KB
            Fragment 2540ad2fb916785d:17301f8a00000001: Reservation=0 OtherMemory=0 Total=0 Peak=160.21 MB
              AGGREGATION_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=160.02 MB
              HDFS_SCAN_NODE (id=1): Total=0 Peak=157.00 KB
              DataStreamSender (dst_id=5): Total=0 Peak=25.59 KB
              CodeGen: Total=0 Peak=417.50 KB
          Query(b94a87085426982d:641c5a9600000000): Reservation=0 ReservationLimit=6.68 GB OtherMemory=0 Total=0 Peak=0
        Untracked Memory: Total=-266538560.00 B
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                mjacobs Matthew Jacobs
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: