Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7010

Multiple flaky tests failing with MemLimitExceeded on S3

    XMLWordPrintableJSON

Details

    • ghx-label-7

    Description

      test_low_mem_limit_orderby_all

      Error Message
      query_test/test_mem_usage_scaling.py:272: in test_low_mem_limit_orderby_all     self.run_primitive_query(vector, 'primitive_orderby_all') query_test/test_mem_usage_scaling.py:260: in run_primitive_query     self.low_memory_limit_test(vector, query_name, self.MIN_MEM[query_name]) query_test/test_mem_usage_scaling.py:114: in low_memory_limit_test     self.run_test_case(tpch_query, new_vector) common/impala_test_suite.py:405: in run_test_case     result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:620: in __execute_query     return impalad_client.execute(query, user=user) common/impala_connection.py:160: in execute     return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:173: in execute     handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:341: in __execute_query     self.wait_for_completion(handle) beeswax/impala_beeswax.py:361: in wait_for_completion     raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E   ImpalaBeeswaxException: ImpalaBeeswaxException: E    Query aborted:Memory limit exceeded: Failed to allocate tuple buffer E   HDFS_SCAN_NODE (id=0) could not allocate 190.00 KB without exceeding limit. E   Error occurred on backend ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment db44c56dcd2fce95:7d746e0800000003 E   Memory left in process limit: 11.40 GB E   Memory left in query limit: 51.61 KB E   Query(db44c56dcd2fce95:7d746e0800000000): Limit=200.00 MB Reservation=158.50 MB ReservationLimit=160.00 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB E     Fragment db44c56dcd2fce95:7d746e0800000003: Reservation=158.50 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB E       SORT_NODE (id=1): Reservation=9.00 MB OtherMemory=8.00 KB Total=9.01 MB Peak=22.31 MB E       HDFS_SCAN_NODE (id=0): Reservation=149.50 MB OtherMemory=41.43 MB Total=190.93 MB Peak=192.13 MB E         Exprs: Total=4.00 KB Peak=4.00 KB E       KrpcDataStreamSender (dst_id=4): Total=688.00 B Peak=688.00 B E       CodeGen: Total=7.72 KB Peak=973.50 KB E    E   Memory limit exceeded: Failed to allocate tuple buffer E   HDFS_SCAN_NODE (id=0) could not allocate 190.00 KB without exceeding limit. E   Error occurred on backend ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment db44c56dcd2fce95:7d746e0800000003 E   Memory left in process limit: 11.40 GB E   Memory left in query limit: 51.61 KB E   Query(db44c56dcd2fce95:7d746e0800000000): Limit=200.00 MB Reservation=158.50 MB ReservationLimit=160.00 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB E     Fragment db44c56dcd2fce95:7d746e0800000003: Reservation=158.50 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB E       SORT_NODE (id=1): Reservation=9.00 MB OtherMemory=8.00 KB Total=9.01 MB Peak=22.31 MB E       HDFS_SCAN_NODE (id=0): Reservation=149.50 MB OtherMemory=41.43 MB Total=190.93 MB Peak=192.13 MB E         Exprs: Total=4.00 KB Peak=4.00 KB E       KrpcDataStreamSender (dst_id=4): Total=688.00 B Peak=688.00 B E       CodeGen: Total=7.72 KB Peak=973.50 KB (1 of 3 similar)
      
      Stacktrace
      query_test/test_mem_usage_scaling.py:272: in test_low_mem_limit_orderby_all
          self.run_primitive_query(vector, 'primitive_orderby_all')
      query_test/test_mem_usage_scaling.py:260: in run_primitive_query
          self.low_memory_limit_test(vector, query_name, self.MIN_MEM[query_name])
      query_test/test_mem_usage_scaling.py:114: in low_memory_limit_test
          self.run_test_case(tpch_query, new_vector)
      common/impala_test_suite.py:405: in run_test_case
          result = self.__execute_query(target_impalad_client, query, user=user)
      common/impala_test_suite.py:620: in __execute_query
          return impalad_client.execute(query, user=user)
      common/impala_connection.py:160: in execute
          return self.__beeswax_client.execute(sql_stmt, user=user)
      beeswax/impala_beeswax.py:173: in execute
          handle = self.__execute_query(query_string.strip(), user=user)
      beeswax/impala_beeswax.py:341: in __execute_query
          self.wait_for_completion(handle)
      beeswax/impala_beeswax.py:361: in wait_for_completion
          raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
      E   ImpalaBeeswaxException: ImpalaBeeswaxException:
      E    Query aborted:Memory limit exceeded: Failed to allocate tuple buffer
      E   HDFS_SCAN_NODE (id=0) could not allocate 190.00 KB without exceeding limit.
      E   Error occurred on backend ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment db44c56dcd2fce95:7d746e0800000003
      E   Memory left in process limit: 11.40 GB
      E   Memory left in query limit: 51.61 KB
      E   Query(db44c56dcd2fce95:7d746e0800000000): Limit=200.00 MB Reservation=158.50 MB ReservationLimit=160.00 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB
      E     Fragment db44c56dcd2fce95:7d746e0800000003: Reservation=158.50 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB
      E       SORT_NODE (id=1): Reservation=9.00 MB OtherMemory=8.00 KB Total=9.01 MB Peak=22.31 MB
      E       HDFS_SCAN_NODE (id=0): Reservation=149.50 MB OtherMemory=41.43 MB Total=190.93 MB Peak=192.13 MB
      E         Exprs: Total=4.00 KB Peak=4.00 KB
      E       KrpcDataStreamSender (dst_id=4): Total=688.00 B Peak=688.00 B
      E       CodeGen: Total=7.72 KB Peak=973.50 KB
      E   
      E   Memory limit exceeded: Failed to allocate tuple buffer
      E   HDFS_SCAN_NODE (id=0) could not allocate 190.00 KB without exceeding limit.
      E   Error occurred on backend ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment db44c56dcd2fce95:7d746e0800000003
      E   Memory left in process limit: 11.40 GB
      E   Memory left in query limit: 51.61 KB
      E   Query(db44c56dcd2fce95:7d746e0800000000): Limit=200.00 MB Reservation=158.50 MB ReservationLimit=160.00 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB
      E     Fragment db44c56dcd2fce95:7d746e0800000003: Reservation=158.50 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB
      E       SORT_NODE (id=1): Reservation=9.00 MB OtherMemory=8.00 KB Total=9.01 MB Peak=22.31 MB
      E       HDFS_SCAN_NODE (id=0): Reservation=149.50 MB OtherMemory=41.43 MB Total=190.93 MB Peak=192.13 MB
      E         Exprs: Total=4.00 KB Peak=4.00 KB
      E       KrpcDataStreamSender (dst_id=4): Total=688.00 B Peak=688.00 B
      E       CodeGen: Total=7.72 KB Peak=973.50 KB (1 of 3 similar)
      
      Standard Error
      -- executing against localhost:21000
      use tpch_parquet;
      
      SET batch_size=0;
      SET num_nodes=0;
      SET disable_codegen_rows_threshold=0;
      SET disable_codegen=False;
      SET abort_on_error=1;
      SET default_spillable_buffer_size=256k;
      SET exec_single_node_rows_threshold=0;
      SET mem_limit=200m;
      MainThread: Query Name: 
      primitive_orderby_all
      
      -- executing against localhost:21000
      -- Description : Scan a fact table and sort 50Million rows.
      -- Target test case : Order by all columns in the table without returning all
      --   rows to the client.
      SELECT *
      FROM (
        SELECT Rank() OVER (
            ORDER BY l_extendedprice
              ,l_orderkey
              ,l_partkey
              ,l_suppkey
              ,l_linenumber
              ,l_quantity
              ,l_discount
              ,l_tax
              ,l_returnflag
              ,l_linestatus
              ,l_shipdate
              ,l_commitdate
              ,l_receiptdate
              ,l_shipinstruct
              ,l_shipmode
              ,l_comment
            ) AS rank
        FROM lineitem
        WHERE l_shipdate < '1992-05-09'
        ) a
      WHERE rank < 10;
      

      test_low_mem_limit_q7

      Error Message
      query_test/test_mem_usage_scaling.py:173: in test_low_mem_limit_q7     self.low_memory_limit_test(vector, 'tpch-q7', self.MIN_MEM_FOR_TPCH['Q7']) query_test/test_mem_usage_scaling.py:114: in low_memory_limit_test     self.run_test_case(tpch_query, new_vector) common/impala_test_suite.py:405: in run_test_case     result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:620: in __execute_query     return impalad_client.execute(query, user=user) common/impala_connection.py:160: in execute     return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:173: in execute     handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:341: in __execute_query     self.wait_for_completion(handle) beeswax/impala_beeswax.py:361: in wait_for_completion     raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E   ImpalaBeeswaxException: ImpalaBeeswaxException: E    Query aborted:Memory limit exceeded: Failed to allocate tuple buffer E   HDFS_SCAN_NODE (id=1) could not allocate 49.00 KB without exceeding limit. E   Error occurred on backend ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment 554fa389af3a9d36:8cd965ba00000003 E   Memory left in process limit: 11.51 GB E   Memory left in query limit: 4.56 KB E   Query(554fa389af3a9d36:8cd965ba00000000): Limit=160.00 MB Reservation=124.06 MB ReservationLimit=128.00 MB OtherMemory=35.93 MB Total=160.00 MB Peak=160.00 MB E     Unclaimed reservations: Reservation=12.75 MB OtherMemory=0 Total=12.75 MB Peak=62.34 MB E     Fragment 554fa389af3a9d36:8cd965ba00000010: Reservation=0 OtherMemory=68.92 KB Total=68.92 KB Peak=941.04 KB E       SORT_NODE (id=12): Total=0 Peak=0 E       AGGREGATION_NODE (id=20): Total=51.12 KB Peak=51.12 KB E         Exprs: Total=51.12 KB Peak=51.12 KB E       EXCHANGE_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=0 E         KrpcDeferredRpcs: Total=0 Peak=0 E       KrpcDataStreamSender (dst_id=21): Total=2.41 KB Peak=2.41 KB E       CodeGen: Total=7.39 KB Peak=879.50 KB E     Fragment 554fa389af3a9d36:8cd965ba0000000d: Reservation=20.31 MB OtherMemory=23.86 MB Total=44.18 MB Peak=44.18 MB E       Runtime Filter Bank: Reservation=5.00 MB ReservationLimit=5.00 MB OtherMemory=0 Total=5.00 MB Peak=5.00 MB E       AGGREGATION_NODE (id=11): Total=51.12 KB Peak=51.12 KB E         Exprs: Total=51.12 KB Peak=51.12 KB E       HASH_JOIN_NODE (id=10): Reservation=1.94 MB OtherMemory=58.25 KB Total=1.99 MB Peak=2.01 MB E         Exprs: Total=11.12 KB Peak=11.12 KB E         Hash Join Builder (join_node_id=10): Total=7.12 KB Peak=15.12 KB E           Hash Join Builder (join_node_id=10) Exprs: Total=7.12 KB Peak=7.12 KB E       HASH_JOIN_NODE (id=9): Reservation=1.94 MB OtherMemory=46.25 KB Total=1.98 MB Peak=2.00 MB E         Exprs: Total=7.12 KB Peak=7.12 KB E         Hash Join Builder (join_node_id=9): Total=7.12 KB Peak=15.12 KB E           Hash Join Builder (join_node_id=9) Exprs: Total=7.12 KB Peak=7.12 KB E       HASH_JOIN_NODE (id=8): Reservation=4.75 MB OtherMemory=58.25 KB Total=4.81 MB Peak=4.81 MB E         Exprs: Total=13.12 KB Peak=13.12 KB E         Hash Join Builder (join_node_id=8): Total=13.12 KB Peak=21.12 KB E           Hash Join Builder (join_node_id=8) Exprs: Total=13.12 KB Peak=13.12 KB E       HASH_JOIN_NODE (id=7): Reservation=1.94 MB OtherMemory=42.25 KB Total=1.98 MB Peak=1.99 MB E         Exprs: Total=13.12 KB Peak=13.12 KB E         Hash Join Builder (join_node_id=7): Total=13.12 KB Peak=21.12 KB E           Hash Join Builder (join_node_id=7) Exprs: Total=13.12 KB Peak=13.12 KB E       HASH_JOIN_NODE (id=6): Reservation=4.75 MB OtherMemory=42.25 KB Total=4.79 MB Peak=4.79 MB E         Exprs: Total=13.12 KB Peak=13.12 KB E         Hash Join Builder (join_node_id=6): Total=13.12 KB Peak=13.12 KB E           Hash Join Builder (join_node_id=6) Exprs: Total=13.12 KB Peak=13.12 KB E       EXCHANGE_NODE (id=13): Reservation=12.66 MB OtherMemory=0 Total=12.66 MB Peak=12.66 MB E         KrpcDeferredRpcs: Total=0 Peak=0 E       EXCHANGE_NODE (id=14): Reservation=7.36 MB OtherMemory=0 Total=7.36 MB Peak=7.36 MB E         KrpcDeferredRpcs: Total=0 Peak=0 E       EXCHANGE_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=240.00 KB E         KrpcDeferredRpcs: Total=0 Peak=0 E       EXCHANGE_NODE (id=16): Reservation=3.44 MB OtherMemory=0 Total=3.44 MB Peak=3.44 MB E         KrpcDeferredRpcs: Total=0 Peak=0 E       EXCHANGE_NODE (id=17): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB E         KrpcDeferredRpcs: Total=0 Peak=0 E       EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB E         KrpcDeferredRpcs: Total=0 Peak=0 E       KrpcDataStreamSender (dst_id=19): Total=7.24 KB Peak=7.24 KB E       CodeGen: Total=107.08 KB Peak=10.86 MB E     Fragment 554fa389af3a9d36:8cd965ba00000003: Reservation=91.00 MB OtherMemory=12.00 MB Total=103.00 MB Peak=103.00 MB E       Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB E       HDFS_SCAN_NODE (id=1): Reservation=89.00 MB OtherMemory=11.76 MB Total=100.76 MB Peak=100.76 MB E         Exprs: Total=4.00 KB Peak=4.00 KB E       KrpcDataStreamSender (dst_id=13): Total=91.83 KB Peak=91.83 KB E       CodeGen: Total=3.33 KB Peak=557.00 KB E     Fragment 554fa389af3a9d36:8cd965ba00000006: Reservation=0 OtherMemory=0 Total=0 Peak=11.37 MB E       HDFS_SCAN_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=10.25 MB E       KrpcDataStreamSender (dst_id=14): Total=0 Peak=106.57 KB E       CodeGen: Total=0 Peak=218.00 KB E     Fragment 554fa389af3a9d36:8cd965ba00000007: Reservation=0 OtherMemory=0 Total=0 Peak=1.43 MB E       HDFS_SCAN_NODE (id=0): Reservation=0 OtherMemory=0 Total=0 Peak=385.42 KB E       KrpcDataStreamSender (dst_id=15): Total=0 Peak=34.90 KB E       CodeGen: Total=0 Peak=220.00 KB E     Fragment 554fa389af3a9d36:8cd965ba00000008: Reservation=0 OtherMemory=0 Total=0 Peak=4.63 MB E       HDFS_SCAN_NODE (id=3): Reservation=0 OtherMemory=0 Total=0 Peak=3.58 MB E       KrpcDataStreamSender (dst_id=16): Total=0 Peak=34.90 KB E       CodeGen: Total=0 Peak=220.00 KB E     Fragment 554fa389af3a9d36:8cd965ba00000009: Reservation=0 OtherMemory=0 Total=0 Peak=95.68 KB E       HDFS_SCAN_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=67.00 KB E       KrpcDataStreamSender (dst_id=17): Total=0 Peak=20.20 KB E       CodeGen: Total=0 Peak=51.00 KB E     Fragment 554fa389af3a9d36:8cd965ba0000000a: Reservation=0 OtherMemory=0 Total=0 Peak=95.68 KB E       HDFS_SCAN_NODE (id=5): Reservation=0 OtherMemory=0 Total=0 Peak=67.00 KB E       KrpcDataStreamSender (dst_id=18): Total=0 Peak=20.20 KB E       CodeGen: Total=0 Peak=51.00 KB E    E   Memory limit exceeded: Failed to allocate tuple buffer E   HDFS_SCAN_NODE (id=1) could not allocate 49.00 KB without exceeding limit. E   Error occurred on backend ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment 554fa389af3a9d36:8cd965ba00000003 E   Memory left in process limit: 11.51 GB E   Memory left in query limit: 4.56 KB E   Query(554fa389af3a9d36:8cd965ba00000000): Limit=160.00 MB Reservation=124.06 MB ReservationLimit=128.00 MB OtherMemory=35.93 MB Total=160.00 MB Peak=160.00 MB E     Unclaimed reservations: Reservation=12.75 MB OtherMemory=0 Total=12.75 MB Peak=62.34 MB E     Fragment 554fa389af3a9d36:8cd965ba00000010: Reservation=0 OtherMemory=68.92 KB Total=68.92 KB Peak=941.04 KB E       SORT_NODE (id=12): Total=0 Peak=0 E       AGGREGATION_NODE (id=20): Total=51.12 KB Peak=51.12 KB E         Exprs: Total=51.12 KB Peak=51.12 KB E       EXCHANGE_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=0 E         KrpcDeferredRpcs: Total=0 Peak=0 E       KrpcDataStreamSender (dst_id=21): Total=2.41 KB Peak=2.41 KB E       CodeGen: Total=7.39 KB Peak=879.50 KB E     Fragment 554fa389af3a9d36:8cd965ba0000000d: Reservation=20.31 MB OtherMemory=23.86 MB Total=44.18 MB Peak=44.18 MB E       Runtime Filter Bank: Reservation=5.00 MB ReservationLimit=5.00 MB OtherMemory=0 Total=5.00 MB Peak=5.00 MB E       AGGREGATION_NODE (id=11): Total=51.12 KB Peak=51.12 KB E         Exprs: Total=51.12 KB Peak=51.12 KB E       HASH_JOIN_NODE (id=10): Reservation=1.94 MB OtherMemory=58.25 KB Total=1.99 MB Peak=2.01 MB E         Exprs: Total=11.12 KB Peak=11.12 KB E         Hash Join Builder (join_node_id=10): Total=7.12 KB Peak=15.12 KB E           Hash Join Builder (join_node_id=10) Exprs: Total=7.12 KB Peak=7.12 KB E       HASH_JOIN_NODE (id=9): Reservation=1.94 MB OtherMemory=46.25 KB Total=1.98 MB Peak=2.00 MB E         Exprs: Total=7.12 KB Peak=7.12 KB E         Hash Join Builder (join_node_id=9): Total=7.12 KB Peak=15.12 KB E           Hash Join Builder (join_node_id=9) Exprs: Total=7.12 KB Peak=7.12 KB E       HASH_JOIN_NODE (id=8): Reservation=4.75 MB OtherMemory=58.25 KB Total=4.81 MB Peak=4.81 MB E         Exprs: Total=13.12 KB Peak=13.12 KB E         Hash Join Builder (join_node_id=8): Total=13.12 KB Peak=21.12 KB E           Hash Join Builder (join_node_id=8) Exprs: Total=13.12 KB Peak=13.12 KB E       HASH_JOIN_NODE (id=7): Reservation=1.94 MB OtherMemory=42.25 KB Total=1.98 MB Peak=1.99 MB E         Exprs: Total=13.12 KB Peak=13.12 KB E         Hash Join Builder (join_node_id=7): Total=13.12 KB Peak=21.12 KB E           Hash Join Builder (join_node_id=7) Exprs: Total=13.12 KB Peak=13.12 KB E       HASH_JOIN_NODE (id=6): Reservation=4.75 MB OtherMemory=42.25 KB Total=4.79 MB Peak=4.79 MB E         Exprs: Total=13.12 KB Peak=13.12 KB E         Hash Join Builder (join_node_id=6): Total=13.12 KB Peak=13.12 KB E           Hash Join Builder (join_node_id=6) Exprs: Total=13.12 KB Peak=13.12 KB E       EXCHANGE_NODE (id=13): Reservation=12.66 MB OtherMemory=0 Total=12.66 MB Peak=12.66 MB E         KrpcDeferredRpcs: Total=0 Peak=0 E       EXCHANGE_NODE (id=14): Reservation=7.36 MB OtherMemory=0 Total=7.36 MB Peak=7.36 MB E         KrpcDeferredRpcs: Total=0 Peak=0 E       EXCHANGE_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=240.00 KB E         KrpcDeferredRpcs: Total=0 Peak=0 E       EXCHANGE_NODE (id=16): Reservation=3.44 MB OtherMemory=0 Total=3.44 MB Peak=3.44 MB E         KrpcDeferredRpcs: Total=0 Peak=0 E       EXCHANGE_NODE (id=17): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB E         KrpcDeferredRpcs: Total=0 Peak=0 E       EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB E         KrpcDeferredRpcs: Total=0 Peak=0 E       KrpcDataStreamSender (dst_id=19): Total=7.24 KB Peak=7.24 KB E       CodeGen: Total=107.08 KB Peak=10.86 MB E     Fragment 554fa389af3a9d36:8cd965ba00000003: Reservation=91.00 MB OtherMemory=12.00 MB Total=103.00 MB Peak=103.00 MB E       Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB E       HDFS_SCAN_NODE (id=1): Reservation=89.00 MB OtherMemory=11.76 MB Total=100.76 MB Peak=100.76 MB E         Exprs: Total=4.00 KB Peak=4.00 KB E       KrpcDataStreamSender (dst_id=13): Total=91.83 KB Peak=91.83 KB E       CodeGen: Total=3.33 KB Peak=557.00 KB E     Fragment 554fa389af3a9d36:8cd965ba00000006: Reservation=0 OtherMemory=0 Total=0 Peak=11.37 MB E       HDFS_SCAN_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=10.25 MB E       KrpcDataStreamSender (dst_id=14): Total=0 Peak=106.57 KB E       CodeGen: Total=0 Peak=218.00 KB E     Fragment 554fa389af3a9d36:8cd965ba00000007: Reservation=0 OtherMemory=0 Total=0 Peak=1.43 MB E       HDFS_SCAN_NODE (id=0): Reservation=0 OtherMemory=0 Total=0 Peak=385.42 KB E       KrpcDataStreamSender (dst_id=15): Total=0 Peak=34.90 KB E       CodeGen: Total=0 Peak=220.00 KB E     Fragment 554fa389af3a9d36:8cd965ba00000008: Reservation=0 OtherMemory=0 Total=0 Peak=4.63 MB E       HDFS_SCAN_NODE (id=3): Reservation=0 OtherMemory=0 Total=0 Peak=3.58 MB E       KrpcDataStreamSender (dst_id=16): Total=0 Peak=34.90 KB E       CodeGen: Total=0 Peak=220.00 KB E     Fragment 554fa389af3a9d36:8cd965ba00000009: Reservation=0 OtherMemory=0 Total=0 Peak=95.68 KB E       HDFS_SCAN_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=67.00 KB E       KrpcDataStreamSender (dst_id=17): Total=0 Peak=20.20 KB E       CodeGen: Total=0 Peak=51.00 KB E     Fragment 554fa389af3a9d36:8cd965ba0000000a: Reservation=0 OtherMemory=0 Total=0 Peak=95.68 KB E       HDFS_SCAN_NODE (id=5): Reservation=0 OtherMemory=0 Total=0 Peak=67.00 KB E       KrpcDataStreamSender (dst_id=18): Total=0 Peak=20.20 KB E       CodeGen: Total=0 Peak=51.00 KB
      
      Stacktrace
      query_test/test_mem_usage_scaling.py:173: in test_low_mem_limit_q7
          self.low_memory_limit_test(vector, 'tpch-q7', self.MIN_MEM_FOR_TPCH['Q7'])
      query_test/test_mem_usage_scaling.py:114: in low_memory_limit_test
          self.run_test_case(tpch_query, new_vector)
      common/impala_test_suite.py:405: in run_test_case
          result = self.__execute_query(target_impalad_client, query, user=user)
      common/impala_test_suite.py:620: in __execute_query
          return impalad_client.execute(query, user=user)
      common/impala_connection.py:160: in execute
          return self.__beeswax_client.execute(sql_stmt, user=user)
      beeswax/impala_beeswax.py:173: in execute
          handle = self.__execute_query(query_string.strip(), user=user)
      beeswax/impala_beeswax.py:341: in __execute_query
          self.wait_for_completion(handle)
      beeswax/impala_beeswax.py:361: in wait_for_completion
          raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
      E   ImpalaBeeswaxException: ImpalaBeeswaxException:
      E    Query aborted:Memory limit exceeded: Failed to allocate tuple buffer
      E   HDFS_SCAN_NODE (id=1) could not allocate 49.00 KB without exceeding limit.
      E   Error occurred on backend ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment 554fa389af3a9d36:8cd965ba00000003
      E   Memory left in process limit: 11.51 GB
      E   Memory left in query limit: 4.56 KB
      E   Query(554fa389af3a9d36:8cd965ba00000000): Limit=160.00 MB Reservation=124.06 MB ReservationLimit=128.00 MB OtherMemory=35.93 MB Total=160.00 MB Peak=160.00 MB
      E     Unclaimed reservations: Reservation=12.75 MB OtherMemory=0 Total=12.75 MB Peak=62.34 MB
      E     Fragment 554fa389af3a9d36:8cd965ba00000010: Reservation=0 OtherMemory=68.92 KB Total=68.92 KB Peak=941.04 KB
      E       SORT_NODE (id=12): Total=0 Peak=0
      E       AGGREGATION_NODE (id=20): Total=51.12 KB Peak=51.12 KB
      E         Exprs: Total=51.12 KB Peak=51.12 KB
      E       EXCHANGE_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=0
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       KrpcDataStreamSender (dst_id=21): Total=2.41 KB Peak=2.41 KB
      E       CodeGen: Total=7.39 KB Peak=879.50 KB
      E     Fragment 554fa389af3a9d36:8cd965ba0000000d: Reservation=20.31 MB OtherMemory=23.86 MB Total=44.18 MB Peak=44.18 MB
      E       Runtime Filter Bank: Reservation=5.00 MB ReservationLimit=5.00 MB OtherMemory=0 Total=5.00 MB Peak=5.00 MB
      E       AGGREGATION_NODE (id=11): Total=51.12 KB Peak=51.12 KB
      E         Exprs: Total=51.12 KB Peak=51.12 KB
      E       HASH_JOIN_NODE (id=10): Reservation=1.94 MB OtherMemory=58.25 KB Total=1.99 MB Peak=2.01 MB
      E         Exprs: Total=11.12 KB Peak=11.12 KB
      E         Hash Join Builder (join_node_id=10): Total=7.12 KB Peak=15.12 KB
      E           Hash Join Builder (join_node_id=10) Exprs: Total=7.12 KB Peak=7.12 KB
      E       HASH_JOIN_NODE (id=9): Reservation=1.94 MB OtherMemory=46.25 KB Total=1.98 MB Peak=2.00 MB
      E         Exprs: Total=7.12 KB Peak=7.12 KB
      E         Hash Join Builder (join_node_id=9): Total=7.12 KB Peak=15.12 KB
      E           Hash Join Builder (join_node_id=9) Exprs: Total=7.12 KB Peak=7.12 KB
      E       HASH_JOIN_NODE (id=8): Reservation=4.75 MB OtherMemory=58.25 KB Total=4.81 MB Peak=4.81 MB
      E         Exprs: Total=13.12 KB Peak=13.12 KB
      E         Hash Join Builder (join_node_id=8): Total=13.12 KB Peak=21.12 KB
      E           Hash Join Builder (join_node_id=8) Exprs: Total=13.12 KB Peak=13.12 KB
      E       HASH_JOIN_NODE (id=7): Reservation=1.94 MB OtherMemory=42.25 KB Total=1.98 MB Peak=1.99 MB
      E         Exprs: Total=13.12 KB Peak=13.12 KB
      E         Hash Join Builder (join_node_id=7): Total=13.12 KB Peak=21.12 KB
      E           Hash Join Builder (join_node_id=7) Exprs: Total=13.12 KB Peak=13.12 KB
      E       HASH_JOIN_NODE (id=6): Reservation=4.75 MB OtherMemory=42.25 KB Total=4.79 MB Peak=4.79 MB
      E         Exprs: Total=13.12 KB Peak=13.12 KB
      E         Hash Join Builder (join_node_id=6): Total=13.12 KB Peak=13.12 KB
      E           Hash Join Builder (join_node_id=6) Exprs: Total=13.12 KB Peak=13.12 KB
      E       EXCHANGE_NODE (id=13): Reservation=12.66 MB OtherMemory=0 Total=12.66 MB Peak=12.66 MB
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       EXCHANGE_NODE (id=14): Reservation=7.36 MB OtherMemory=0 Total=7.36 MB Peak=7.36 MB
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       EXCHANGE_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=240.00 KB
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       EXCHANGE_NODE (id=16): Reservation=3.44 MB OtherMemory=0 Total=3.44 MB Peak=3.44 MB
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       EXCHANGE_NODE (id=17): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       KrpcDataStreamSender (dst_id=19): Total=7.24 KB Peak=7.24 KB
      E       CodeGen: Total=107.08 KB Peak=10.86 MB
      E     Fragment 554fa389af3a9d36:8cd965ba00000003: Reservation=91.00 MB OtherMemory=12.00 MB Total=103.00 MB Peak=103.00 MB
      E       Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
      E       HDFS_SCAN_NODE (id=1): Reservation=89.00 MB OtherMemory=11.76 MB Total=100.76 MB Peak=100.76 MB
      E         Exprs: Total=4.00 KB Peak=4.00 KB
      E       KrpcDataStreamSender (dst_id=13): Total=91.83 KB Peak=91.83 KB
      E       CodeGen: Total=3.33 KB Peak=557.00 KB
      E     Fragment 554fa389af3a9d36:8cd965ba00000006: Reservation=0 OtherMemory=0 Total=0 Peak=11.37 MB
      E       HDFS_SCAN_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=10.25 MB
      E       KrpcDataStreamSender (dst_id=14): Total=0 Peak=106.57 KB
      E       CodeGen: Total=0 Peak=218.00 KB
      E     Fragment 554fa389af3a9d36:8cd965ba00000007: Reservation=0 OtherMemory=0 Total=0 Peak=1.43 MB
      E       HDFS_SCAN_NODE (id=0): Reservation=0 OtherMemory=0 Total=0 Peak=385.42 KB
      E       KrpcDataStreamSender (dst_id=15): Total=0 Peak=34.90 KB
      E       CodeGen: Total=0 Peak=220.00 KB
      E     Fragment 554fa389af3a9d36:8cd965ba00000008: Reservation=0 OtherMemory=0 Total=0 Peak=4.63 MB
      E       HDFS_SCAN_NODE (id=3): Reservation=0 OtherMemory=0 Total=0 Peak=3.58 MB
      E       KrpcDataStreamSender (dst_id=16): Total=0 Peak=34.90 KB
      E       CodeGen: Total=0 Peak=220.00 KB
      E     Fragment 554fa389af3a9d36:8cd965ba00000009: Reservation=0 OtherMemory=0 Total=0 Peak=95.68 KB
      E       HDFS_SCAN_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=67.00 KB
      E       KrpcDataStreamSender (dst_id=17): Total=0 Peak=20.20 KB
      E       CodeGen: Total=0 Peak=51.00 KB
      E     Fragment 554fa389af3a9d36:8cd965ba0000000a: Reservation=0 OtherMemory=0 Total=0 Peak=95.68 KB
      E       HDFS_SCAN_NODE (id=5): Reservation=0 OtherMemory=0 Total=0 Peak=67.00 KB
      E       KrpcDataStreamSender (dst_id=18): Total=0 Peak=20.20 KB
      E       CodeGen: Total=0 Peak=51.00 KB
      E   
      E   Memory limit exceeded: Failed to allocate tuple buffer
      E   HDFS_SCAN_NODE (id=1) could not allocate 49.00 KB without exceeding limit.
      E   Error occurred on backend ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment 554fa389af3a9d36:8cd965ba00000003
      E   Memory left in process limit: 11.51 GB
      E   Memory left in query limit: 4.56 KB
      E   Query(554fa389af3a9d36:8cd965ba00000000): Limit=160.00 MB Reservation=124.06 MB ReservationLimit=128.00 MB OtherMemory=35.93 MB Total=160.00 MB Peak=160.00 MB
      E     Unclaimed reservations: Reservation=12.75 MB OtherMemory=0 Total=12.75 MB Peak=62.34 MB
      E     Fragment 554fa389af3a9d36:8cd965ba00000010: Reservation=0 OtherMemory=68.92 KB Total=68.92 KB Peak=941.04 KB
      E       SORT_NODE (id=12): Total=0 Peak=0
      E       AGGREGATION_NODE (id=20): Total=51.12 KB Peak=51.12 KB
      E         Exprs: Total=51.12 KB Peak=51.12 KB
      E       EXCHANGE_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=0
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       KrpcDataStreamSender (dst_id=21): Total=2.41 KB Peak=2.41 KB
      E       CodeGen: Total=7.39 KB Peak=879.50 KB
      E     Fragment 554fa389af3a9d36:8cd965ba0000000d: Reservation=20.31 MB OtherMemory=23.86 MB Total=44.18 MB Peak=44.18 MB
      E       Runtime Filter Bank: Reservation=5.00 MB ReservationLimit=5.00 MB OtherMemory=0 Total=5.00 MB Peak=5.00 MB
      E       AGGREGATION_NODE (id=11): Total=51.12 KB Peak=51.12 KB
      E         Exprs: Total=51.12 KB Peak=51.12 KB
      E       HASH_JOIN_NODE (id=10): Reservation=1.94 MB OtherMemory=58.25 KB Total=1.99 MB Peak=2.01 MB
      E         Exprs: Total=11.12 KB Peak=11.12 KB
      E         Hash Join Builder (join_node_id=10): Total=7.12 KB Peak=15.12 KB
      E           Hash Join Builder (join_node_id=10) Exprs: Total=7.12 KB Peak=7.12 KB
      E       HASH_JOIN_NODE (id=9): Reservation=1.94 MB OtherMemory=46.25 KB Total=1.98 MB Peak=2.00 MB
      E         Exprs: Total=7.12 KB Peak=7.12 KB
      E         Hash Join Builder (join_node_id=9): Total=7.12 KB Peak=15.12 KB
      E           Hash Join Builder (join_node_id=9) Exprs: Total=7.12 KB Peak=7.12 KB
      E       HASH_JOIN_NODE (id=8): Reservation=4.75 MB OtherMemory=58.25 KB Total=4.81 MB Peak=4.81 MB
      E         Exprs: Total=13.12 KB Peak=13.12 KB
      E         Hash Join Builder (join_node_id=8): Total=13.12 KB Peak=21.12 KB
      E           Hash Join Builder (join_node_id=8) Exprs: Total=13.12 KB Peak=13.12 KB
      E       HASH_JOIN_NODE (id=7): Reservation=1.94 MB OtherMemory=42.25 KB Total=1.98 MB Peak=1.99 MB
      E         Exprs: Total=13.12 KB Peak=13.12 KB
      E         Hash Join Builder (join_node_id=7): Total=13.12 KB Peak=21.12 KB
      E           Hash Join Builder (join_node_id=7) Exprs: Total=13.12 KB Peak=13.12 KB
      E       HASH_JOIN_NODE (id=6): Reservation=4.75 MB OtherMemory=42.25 KB Total=4.79 MB Peak=4.79 MB
      E         Exprs: Total=13.12 KB Peak=13.12 KB
      E         Hash Join Builder (join_node_id=6): Total=13.12 KB Peak=13.12 KB
      E           Hash Join Builder (join_node_id=6) Exprs: Total=13.12 KB Peak=13.12 KB
      E       EXCHANGE_NODE (id=13): Reservation=12.66 MB OtherMemory=0 Total=12.66 MB Peak=12.66 MB
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       EXCHANGE_NODE (id=14): Reservation=7.36 MB OtherMemory=0 Total=7.36 MB Peak=7.36 MB
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       EXCHANGE_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=240.00 KB
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       EXCHANGE_NODE (id=16): Reservation=3.44 MB OtherMemory=0 Total=3.44 MB Peak=3.44 MB
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       EXCHANGE_NODE (id=17): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       KrpcDataStreamSender (dst_id=19): Total=7.24 KB Peak=7.24 KB
      E       CodeGen: Total=107.08 KB Peak=10.86 MB
      E     Fragment 554fa389af3a9d36:8cd965ba00000003: Reservation=91.00 MB OtherMemory=12.00 MB Total=103.00 MB Peak=103.00 MB
      E       Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
      E       HDFS_SCAN_NODE (id=1): Reservation=89.00 MB OtherMemory=11.76 MB Total=100.76 MB Peak=100.76 MB
      E         Exprs: Total=4.00 KB Peak=4.00 KB
      E       KrpcDataStreamSender (dst_id=13): Total=91.83 KB Peak=91.83 KB
      E       CodeGen: Total=3.33 KB Peak=557.00 KB
      E     Fragment 554fa389af3a9d36:8cd965ba00000006: Reservation=0 OtherMemory=0 Total=0 Peak=11.37 MB
      E       HDFS_SCAN_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=10.25 MB
      E       KrpcDataStreamSender (dst_id=14): Total=0 Peak=106.57 KB
      E       CodeGen: Total=0 Peak=218.00 KB
      E     Fragment 554fa389af3a9d36:8cd965ba00000007: Reservation=0 OtherMemory=0 Total=0 Peak=1.43 MB
      E       HDFS_SCAN_NODE (id=0): Reservation=0 OtherMemory=0 Total=0 Peak=385.42 KB
      E       KrpcDataStreamSender (dst_id=15): Total=0 Peak=34.90 KB
      E       CodeGen: Total=0 Peak=220.00 KB
      E     Fragment 554fa389af3a9d36:8cd965ba00000008: Reservation=0 OtherMemory=0 Total=0 Peak=4.63 MB
      E       HDFS_SCAN_NODE (id=3): Reservation=0 OtherMemory=0 Total=0 Peak=3.58 MB
      E       KrpcDataStreamSender (dst_id=16): Total=0 Peak=34.90 KB
      E       CodeGen: Total=0 Peak=220.00 KB
      E     Fragment 554fa389af3a9d36:8cd965ba00000009: Reservation=0 OtherMemory=0 Total=0 Peak=95.68 KB
      E       HDFS_SCAN_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=67.00 KB
      E       KrpcDataStreamSender (dst_id=17): Total=0 Peak=20.20 KB
      E       CodeGen: Total=0 Peak=51.00 KB
      E     Fragment 554fa389af3a9d36:8cd965ba0000000a: Reservation=0 OtherMemory=0 Total=0 Peak=95.68 KB
      E       HDFS_SCAN_NODE (id=5): Reservation=0 OtherMemory=0 Total=0 Peak=67.00 KB
      E       KrpcDataStreamSender (dst_id=18): Total=0 Peak=20.20 KB
      E       CodeGen: Total=0 Peak=51.00 KB
      
      Standard Error
      -- executing against localhost:21000
      use tpch_parquet;
      
      SET batch_size=0;
      SET num_nodes=0;
      SET disable_codegen_rows_threshold=0;
      SET disable_codegen=False;
      SET abort_on_error=1;
      SET default_spillable_buffer_size=256k;
      SET exec_single_node_rows_threshold=0;
      SET mem_limit=160m;
      MainThread: Query Name: 
      TPCH-Q7
      
      -- executing against localhost:21000
      select
        supp_nation,
        cust_nation,
        l_year,
        sum(volume) as revenue
      from (
        select
          n1.n_name as supp_nation,
          n2.n_name as cust_nation,
          year(l_shipdate) as l_year,
          l_extendedprice * (1 - l_discount) as volume
        from
          supplier,
          lineitem,
          orders,
          customer,
          nation n1,
          nation n2
        where
          s_suppkey = l_suppkey
          and o_orderkey = l_orderkey
          and c_custkey = o_custkey
          and s_nationkey = n1.n_nationkey
          and c_nationkey = n2.n_nationkey
          and (
            (n1.n_name = 'FRANCE' and n2.n_name = 'GERMANY')
            or (n1.n_name = 'GERMANY' and n2.n_name = 'FRANCE')
          )
          and l_shipdate between '1995-01-01' and '1996-12-31'
        ) as shipping
      group by
        supp_nation,
        cust_nation,
        l_year
      order by
        supp_nation,
        cust_nation,
        l_year;
      

      Attachments

        Activity

          People

            tarmstrong Tim Armstrong
            sailesh Sailesh Mukil
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: