Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5773

Memory limit exceeded on test_spilling.py

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0
    • Fix Version/s: Impala 2.10.0
    • Component/s: Backend
    • Labels:

      Description

      03:55:47 FAIL query_test/test_spilling.py::TestSpilling::()::test_spilling[exec_option: {'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
      03:55:47 =================================== FAILURES ===================================
      03:55:47  TestSpilling.test_spilling[exec_option: {'default_spillable_buffer_size': '256k'} | table_format: parquet/none] 
      03:55:47 [gw1] linux2 -- Python 2.6.6 /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/bin/../infra/python/env/bin/python
      03:55:47 query_test/test_spilling.py:39: in test_spilling
      03:55:47     self.run_test_case('QueryTest/spilling', vector)
      03:55:47 common/impala_test_suite.py:390: in run_test_case
      03:55:47     result = self.__execute_query(target_impalad_client, query, user=user)
      03:55:47 common/impala_test_suite.py:598: in __execute_query
      03:55:47     return impalad_client.execute(query, user=user)
      03:55:47 common/impala_connection.py:160: in execute
      03:55:47     return self.__beeswax_client.execute(sql_stmt, user=user)
      03:55:47 beeswax/impala_beeswax.py:173: in execute
      03:55:47     handle = self.__execute_query(query_string.strip(), user=user)
      03:55:47 beeswax/impala_beeswax.py:339: in __execute_query
      03:55:47     self.wait_for_completion(handle)
      03:55:47 beeswax/impala_beeswax.py:359: in wait_for_completion
      03:55:47     raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
      03:55:47 E   ImpalaBeeswaxException: ImpalaBeeswaxException:
      03:55:47 E    Query aborted:Memory limit exceeded: Error occurred on backend impala-boost-static-burst-slave-1c37.vpc.cloudera.com:22000 by fragment be415e2081bde55d:ce5cb0b400000001
      03:55:47 E   Memory left in process limit: 49.35 GB
      03:55:47 E   Memory left in query limit: -36546.00 B
      03:55:47 E   Query(be415e2081bde55d:ce5cb0b400000000): memory limit exceeded. Limit=800.00 MB Reservation=640.00 MB ReservationLimit=640.00 MB OtherMemory=160.03 MB Total=800.03 MB Peak=800.03 MB
      03:55:47 E     Fragment be415e2081bde55d:ce5cb0b400000000: Reservation=0 OtherMemory=12.24 KB Total=12.24 KB Peak=63.50 KB
      03:55:47 E       AGGREGATION_NODE (id=6): Total=4.00 KB Peak=4.00 KB
      03:55:47 E         Exprs: Total=4.00 KB Peak=4.00 KB
      03:55:47 E       EXCHANGE_NODE (id=5): Total=0 Peak=0
      03:55:47 E       DataStreamRecvr: Total=0 Peak=0
      03:55:47 E       PLAN_ROOT_SINK: Total=0 Peak=0
      03:55:47 E       CodeGen: Total=247.00 B Peak=51.50 KB
      03:55:47 E     Fragment be415e2081bde55d:ce5cb0b400000002: Reservation=24.50 MB OtherMemory=157.98 MB Total=182.48 MB Peak=182.48 MB
      03:55:47 E       AGGREGATION_NODE (id=2): Total=4.00 KB Peak=4.00 KB
      03:55:47 E         Exprs: Total=4.00 KB Peak=4.00 KB
      03:55:47 E       AGGREGATION_NODE (id=4): Reservation=24.50 MB OtherMemory=29.12 KB Total=24.53 MB Peak=25.80 MB
      03:55:47 E         Exprs: Total=21.12 KB Peak=21.12 KB
      03:55:47 E       EXCHANGE_NODE (id=3): Total=0 Peak=0
      03:55:47 E       DataStreamRecvr: Total=157.20 MB Peak=157.20 MB
      03:55:47 E       DataStreamSender (dst_id=5): Total=16.00 KB Peak=16.00 KB
      03:55:47 E       CodeGen: Total=5.09 KB Peak=373.50 KB
      03:55:47 E     Fragment be415e2081bde55d:ce5cb0b400000001: Reservation=615.50 MB OtherMemory=2.04 MB Total=617.54 MB Peak=651.88 MB
      03:55:47 E       AGGREGATION_NODE (id=1): Reservation=615.50 MB OtherMemory=2.02 MB Total=617.52 MB Peak=618.70 MB
      03:55:47 E         Exprs: Total=2.02 MB Peak=2.02 MB
      03:55:47 E       HDFS_SCAN_NODE (id=0): Total=0 Peak=33.32 MB
      03:55:47 E       DataStreamSender (dst_id=3): Total=7.52 KB Peak=7.52 KB
      03:55:47 E       CodeGen: Total=6.84 KB Peak=522.50 KB
      03:55:47 E   
      03:55:47 E   Memory limit exceeded: Error occurred on backend impala-boost-static-burst-slave-1c37.vpc.cloudera.com:22000 by fragment be415e2081bde55d:ce5cb0b400000001
      03:55:47 E   Memory left in process limit: 49.35 GB
      03:55:47 E   Memory left in query limit: -36546.00 B
      03:55:47 E   Query(be415e2081bde55d:ce5cb0b400000000): memory limit exceeded. Limit=800.00 MB Reservation=640.00 MB ReservationLimit=640.00 MB OtherMemory=160.03 MB Total=800.03 MB Peak=800.03 MB
      03:55:47 E     Fragment be415e2081bde55d:ce5cb0b400000000: Reservation=0 OtherMemory=12.24 KB Total=12.24 KB Peak=63.50 KB
      03:55:47 E       AGGREGATION_NODE (id=6): Total=4.00 KB Peak=4.00 KB
      03:55:47 E         Exprs: Total=4.00 KB Peak=4.00 KB
      03:55:47 E       EXCHANGE_NODE (id=5): Total=0 Peak=0
      03:55:47 E       DataStreamRecvr: Total=0 Peak=0
      03:55:47 E       PLAN_ROOT_SINK: Total=0 Peak=0
      03:55:47 E       CodeGen: Total=247.00 B Peak=51.50 KB
      03:55:47 E     Fragment be415e2081bde55d:ce5cb0b400000002: Reservation=24.50 MB OtherMemory=157.98 MB Total=182.48 MB Peak=182.48 MB
      03:55:47 E       AGGREGATION_NODE (id=2): Total=4.00 KB Peak=4.00 KB
      03:55:47 E         Exprs: Total=4.00 KB Peak=4.00 KB
      03:55:47 E       AGGREGATION_NODE (id=4): Reservation=24.50 MB OtherMemory=29.12 KB Total=24.53 MB Peak=25.80 MB
      03:55:47 E         Exprs: Total=21.12 KB Peak=21.12 KB
      03:55:47 E       EXCHANGE_NODE (id=3): Total=0 Peak=0
      03:55:47 E       DataStreamRecvr: Total=157.20 MB Peak=157.20 MB
      03:55:47 E       DataStreamSender (dst_id=5): Total=16.00 KB Peak=16.00 KB
      03:55:47 E       CodeGen: Total=5.09 KB Peak=373.50 KB
      03:55:47 E     Fragment be415e2081bde55d:ce5cb0b400000001: Reservation=615.50 MB OtherMemory=2.04 MB Total=617.54 MB Peak=651.88 MB
      03:55:47 E       AGGREGATION_NODE (id=1): Reservation=615.50 MB OtherMemory=2.02 MB Total=617.52 MB Peak=618.70 MB
      03:55:47 E         Exprs: Total=2.02 MB Peak=2.02 MB
      03:55:47 E       HDFS_SCAN_NODE (id=0): Total=0 Peak=33.32 MB
      03:55:47 E       DataStreamSender (dst_id=3): Total=7.52 KB Peak=7.52 KB
      03:55:47 E       CodeGen: Total=6.84 KB Peak=522.50 KB (1 of 2 similar)
      03:55:47 ---------------------------- Captured stderr setup -----------------------------
      03:55:47 -- connecting to: localhost:21000
      03:55:47 ----------------------------- Captured stderr call -----------------------------
      03:55:47 -- executing against localhost:21000
      03:55:47 use tpch_parquet;
      03:55:47 
      03:55:47 SET default_spillable_buffer_size=256k;
      03:55:47 -- executing against localhost:21000
      03:55:47 set buffer_pool_limit=10m;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 select l_orderkey, count(*)
      03:55:47 from lineitem
      03:55:47 group by 1
      03:55:47 order by 1 limit 10;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET BUFFER_POOL_LIMIT=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 set buffer_pool_limit=10m;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 set num_nodes=1;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 select l_returnflag, l_orderkey, avg(l_tax), min(l_shipmode)
      03:55:47 from lineitem
      03:55:47 group by 1,2
      03:55:47 order by 1,2 limit 3;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET BUFFER_POOL_LIMIT=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET NUM_NODES=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 set buffer_pool_limit=10m;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 select l_orderkey, count(*)
      03:55:47 from lineitem
      03:55:47 group by 1
      03:55:47 order by 1 limit 10;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET BUFFER_POOL_LIMIT=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 set buffer_pool_limit=10m;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 set num_nodes=1;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 select l_comment, count(*)
      03:55:47 from lineitem
      03:55:47 group by 1
      03:55:47 order by count(*) desc limit 5;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET BUFFER_POOL_LIMIT=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET NUM_NODES=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 set buffer_pool_limit=10m;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 set num_nodes=1;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 select l_returnflag, l_orderkey, round(avg(l_tax),2), min(l_shipmode)
      03:55:47 from lineitem
      03:55:47 group by 1,2
      03:55:47 order by 1,2 limit 3;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET BUFFER_POOL_LIMIT=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET NUM_NODES=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 set buffer_pool_limit=10m;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 select l_orderkey, avg(l_orderkey)
      03:55:47 from lineitem
      03:55:47 group by 1
      03:55:47 order by 1 limit 5;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET BUFFER_POOL_LIMIT=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 set buffer_pool_limit=15m;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 select count(l1.l_tax)
      03:55:47 from
      03:55:47 lineitem l1,
      03:55:47 lineitem l2,
      03:55:47 lineitem l3
      03:55:47 where
      03:55:47 l1.l_tax < 0.01 and
      03:55:47 l2.l_tax < 0.04 and
      03:55:47 l1.l_orderkey = l2.l_orderkey and
      03:55:47 l1.l_orderkey = l3.l_orderkey and
      03:55:47 l1.l_comment = l3.l_comment and
      03:55:47 l1.l_shipdate = l3.l_shipdate;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET BUFFER_POOL_LIMIT=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 set buffer_pool_limit=2m;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 select max(t1.total_count), max(t1.l_shipinstruct), max(t1.l_comment) from
      03:55:47 (select l_shipinstruct, l_comment, count(*) over () total_count from lineitem) t1;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET BUFFER_POOL_LIMIT=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 set buffer_pool_limit=20m;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 select a.int_col, count(*)
      03:55:47 from functional.alltypessmall a, functional.alltypessmall b, functional.alltypessmall c
      03:55:47 where a.id = b.id and b.id = c.id group by a.int_col;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET BUFFER_POOL_LIMIT=0;;
      03:55:47 
      03:55:47 MainThread: Query Name: 
      03:55:47 TPCH-Q22
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 set buffer_pool_limit=20m;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 select
      03:55:47   s_name,
      03:55:47   count(*) as numwait
      03:55:47 from
      03:55:47   supplier,
      03:55:47   lineitem l1 join [BROADCAST]
      03:55:47   orders,
      03:55:47   nation
      03:55:47 where
      03:55:47   s_suppkey = l1.l_suppkey
      03:55:47   and o_orderkey = l1.l_orderkey
      03:55:47   and o_orderstatus = 'F'
      03:55:47   and l1.l_receiptdate > l1.l_commitdate
      03:55:47   and exists (
      03:55:47     select
      03:55:47       *
      03:55:47     from
      03:55:47       lineitem l2
      03:55:47     where
      03:55:47       l2.l_orderkey = l1.l_orderkey
      03:55:47       and l2.l_suppkey <> l1.l_suppkey
      03:55:47   )
      03:55:47   and not exists (
      03:55:47     select
      03:55:47       *
      03:55:47     from
      03:55:47       lineitem l3
      03:55:47     where
      03:55:47       l3.l_orderkey = l1.l_orderkey
      03:55:47       and l3.l_suppkey <> l1.l_suppkey
      03:55:47       and l3.l_receiptdate > l3.l_commitdate
      03:55:47   )
      03:55:47   and s_nationkey = n_nationkey
      03:55:47   and n_name = 'SAUDI ARABIA'
      03:55:47 group by
      03:55:47   s_name
      03:55:47 order by
      03:55:47   numwait desc,
      03:55:47   s_name
      03:55:47 limit 100;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET BUFFER_POOL_LIMIT=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 set buffer_pool_limit=50m;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 select l_orderkey, count(*), group_concat(distinct l_linestatus, '|')
      03:55:47 from lineitem
      03:55:47 group by 1
      03:55:47 order by 1 limit 10;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET BUFFER_POOL_LIMIT=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 set mem_limit=800m;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 set num_scanner_threads=1;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 
      03:55:47 select count(distinct concat(cast(l_comment as char(120)), cast(l_comment as char(120)),
      03:55:47                              cast(l_comment as char(120)), cast(l_comment as char(120)),
      03:55:47                              cast(l_comment as char(120)), cast(l_comment as char(120))))
      03:55:47 from lineitem;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET MEM_LIMIT=0;;
      03:55:47 
      03:55:47 -- executing against localhost:21000
      03:55:47 SET NUM_SCANNER_THREADS=0;;
      03:55:47 
      03:55:47 ======= 1 failed, 1323 passed, 174 skipped, 4 xfailed in 3454.29 seconds =======
      03:55:47 ============================= test session starts ==============================
      

      Tim Armstrong, can you please take a first look ?

        Attachments

          Activity

            People

            • Assignee:
              henryr Henry Robinson
              Reporter:
              kwho Michael Ho
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: