[IMPALA-10251] test_decimal_queries and test_tpcds_queries may run out of memory on ASAN builds

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: Infrastructure
• Labels: broken-build flaky
• Epic Color: ghx-label-7
• Assignee: Unassigned
• Reporter: Quanlong Huang
• Created: 16/Oct/20 08:52
• Updated: 23/Dec/20 19:16

Description

Encountered these test failures due to running out of memory:

query_test.test_decimal_queries.TestDecimalQueries.test_queries[protocol: beeswax | exec_option: {'disable_codegen_rows_threshold': 0, 'disable_codegen': 'false', 'decimal_v2': 'true', 'batch_size': 1} | table_format: orc/def/block]query_test.test_tpcds_queries.TestTpcdsQuery.test_tpcds_q51a[protocol: beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]query_test.test_decimal_queries.TestDecimalQueries.test_queries[protocol: hs2 | exec_option: {'disable_codegen_rows_threshold': 0, 'disable_codegen': 'true', 'decimal_v2': 'true', 'batch_size': 1} | table_format: parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q18a[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q23_1[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q4[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 

The error is

query_test/test_decimal_queries.py:55: in test_queries
    self.run_test_case('QueryTest/decimal', vector)
common/impala_test_suite.py:662: in run_test_case
    result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
common/impala_test_suite.py:600: in __exec_in_impala
    result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:909: in __execute_query
    return impalad_client.execute(query, user=user)
common/impala_connection.py:205: in execute
    return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:187: in execute
    handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:365: in __execute_query
    self.wait_for_finished(handle)
beeswax/impala_beeswax.py:386: in wait_for_finished
    raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E   ImpalaBeeswaxException: ImpalaBeeswaxException:
E    Query aborted:Failed to get minimum memory reservation of 68.01 MB on daemon impala-ec2-centos74-r5-4xlarge-ondemand-109b.vpc.cloudera.com:27001 for query 2b41e9326825c48d:15a4d93100000000 due to following error: Failed to increase reservation by 68.01 MB because it would exceed the applicable reservation limit for the "Process" ReservationTracker: reservation_limit=10.20 GB reservation=10.16 GB used_reservation=0 child_reservations=10.16 GB
E   The top 5 queries that allocated memory under this tracker are:
E   Query(ca4c47a813876f70:d626ca0c00000000): Reservation=9.60 GB ReservationLimit=9.60 GB OtherMemory=115.84 MB Total=9.71 GB Peak=9.71 GB
E   Query(3f4e984e6b54204d:2597d87f00000000): Reservation=426.77 MB ReservationLimit=9.60 GB OtherMemory=113.19 MB Total=539.95 MB Peak=540.00 MB
E   Query(6c4aca5cba7141b7:cb0d9a4000000000): Limit=512.00 MB Reservation=68.03 MB ReservationLimit=409.60 MB OtherMemory=2.59 MB Total=70.62 MB Peak=71.93 MB
E   Query(e84c2ac37bac9309:7eddf8bb00000000): Reservation=14.94 MB ReservationLimit=9.60 GB OtherMemory=5.55 MB Total=20.48 MB Peak=39.46 MB
E   Query(eb44550f4e6aebbd:4b72c39600000000): Reservation=12.31 MB ReservationLimit=9.60 GB OtherMemory=5.85 MB Total=18.16 MB Peak=18.16 MB
E   
E   
E   
E   
E   
E   Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error.