Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5838

Suggested MEM_LIMIT in rejected query error may be too low

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.10.0
    • Fix Version/s: Impala 2.10.0
    • Component/s: Backend
    • Labels:

      Description

      If you set a small memlimit, it may suggest that the memlimit should be at least 79.75 MB. Then, if you run it with 80MB, it'll suggest 92MB.

      [philip-dev.gce.cloudera.com:21000] > set mem_limit=1024;
      MEM_LIMIT set to 1024
      [philip-dev.gce.cloudera.com:21000] > select count(*) from functional.alltypes;
      Query: select count(*) from functional.alltypes
      Query submitted at: 2017-08-24 08:36:37 (Coordinator: http://philip-dev.gce.cloudera.com:25000)
      ERROR: Memory limit exceeded: Could not allocate aggregate expression intermediate value
      Exprs could not allocate 16.00 B without exceeding limit.
      Error occurred on backend philip-dev.gce.cloudera.com:22000 by fragment 704f7351ee03f7f1:5ed9277d00000003
      Memory left in process limit: 31.34 GB
      Memory left in query limit: 1.00 KB
      Query(704f7351ee03f7f1:5ed9277d00000000): Limit=1.00 KB Reservation=0 ReservationLimit=0 OtherMemory=0 Total=0 Peak=0
        Fragment 704f7351ee03f7f1:5ed9277d00000000: Reservation=0 OtherMemory=0 Total=0 Peak=0
          AGGREGATION_NODE (id=3): Total=0 Peak=0
          EXCHANGE_NODE (id=2): Total=0 Peak=0
          DataStreamRecvr: Total=0 Peak=0
        Fragment 704f7351ee03f7f1:5ed9277d00000003: Reservation=0 OtherMemory=0 Total=0 Peak=0
          AGGREGATION_NODE (id=1): Total=0 Peak=0
          HDFS_SCAN_NODE (id=0): Total=0 Peak=0
      
      [philip-dev.gce.cloudera.com:21000] >       select count(*)
            from tpch_parquet.lineitem join tpch_parquet.orders on l_orderkey = o_orderkey;
      Query: select count(*)
            from tpch_parquet.lineitem join tpch_parquet.orders on l_orderkey = o_orderkey
      Query submitted at: 2017-08-24 08:36:46 (Coordinator: http://philip-dev.gce.cloudera.com:25000)
      ERROR: Rejected query from pool default-pool: minimum memory reservation is greater than memory available to the query for buffer reservations. Mem available for buffer reservations based on mem_limit: 1.00 KB, memory reservation needed: 4.75 MB. Set mem_limit to at least 79.75 MB. See the query profile for more information.
      
      [philip-dev.gce.cloudera.com:21000] > set mem_limit=80mb;
      MEM_LIMIT set to 80mb
      [philip-dev.gce.cloudera.com:21000] >       select count(*)
            from tpch_parquet.lineitem join tpch_parquet.orders on l_orderkey = o_orderkey;
      Query: select count(*)
            from tpch_parquet.lineitem join tpch_parquet.orders on l_orderkey = o_orderkey
      Query submitted at: 2017-08-24 08:37:01 (Coordinator: http://philip-dev.gce.cloudera.com:25000)
      ERROR: Rejected query from pool default-pool: minimum memory reservation is greater than memory available to the query for buffer reservations. Mem available for buffer reservations based on mem_limit: 80.00 MB, memory reservation needed: 17.00 MB. Set mem_limit to at least 92.00 MB. See the query profile for more information.
      

      The join is switching from broadcast to partitioned based on the mem_limit, which then affects the per-node bytes estimate, which then affects the buffer size chosen by the planner, which then affects the reservation. This is the code
      https://github.com/apache/incubator-impala/blob/c7db60aa46565c19634e8a791df3af8d116b9017/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java#L539

      The good news is that it will converge after bumping the mem_limit to the recommended value the second time. This is still very wonky.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mjacobs Matthew Jacobs
                Reporter:
                mjacobs Matthew Jacobs
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: