[IMPALA-5908] Multiple tests fail ASAN build due to admission control rejection

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0
• Component/s: Infrastructure
• Labels: broken-build
• Target Version: Impala 2.11.0
• Epic Color: ghx-label-4
• Assignee: Philip Martin
• Reporter: Sailesh Mukil
• Created: 07/Sep/17 18:18
• Updated: 06/Oct/17 18:54
• Resolved: 06/Oct/17 18:54

Description

On a recent run with the ASAN build, I noticed multiple tests fail due to admission control rejection.

00:51:13  TestQueries.test_analytic_fns[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:51:13 [gw3] linux2 -- Python 2.6.6 /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/bin/../infra/python/env/bin/python
00:51:13 query_test/test_queries.py:62: in test_analytic_fns
00:51:13     self.run_test_case('QueryTest/analytic-fns', vector)
00:51:13 common/impala_test_suite.py:390: in run_test_case
00:51:13     result = self.__execute_query(target_impalad_client, query, user=user)
00:51:13 common/impala_test_suite.py:598: in __execute_query
00:51:13     return impalad_client.execute(query, user=user)
00:51:13 common/impala_connection.py:160: in execute
00:51:13     return self.__beeswax_client.execute(sql_stmt, user=user)
00:51:13 beeswax/impala_beeswax.py:173: in execute
00:51:13     handle = self.__execute_query(query_string.strip(), user=user)
00:51:13 beeswax/impala_beeswax.py:339: in __execute_query
00:51:13     handle = self.execute_query_async(query_string, user=user)
00:51:13 beeswax/impala_beeswax.py:335: in execute_query_async
00:51:13     return self.__do_rpc(lambda: self.imp_service.query(query,))
00:51:13 beeswax/impala_beeswax.py:460: in __do_rpc
00:51:13     raise ImpalaBeeswaxException(self.__build_error_message(b), b)
00:51:13 E   ImpalaBeeswaxException: ImpalaBeeswaxException:
00:51:13 E    INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
00:51:13 E    MESSAGE: Rejected query from pool default-pool: minimum memory reservation is greater than memory available to the query for buffer reservations. Increase the buffer_pool_limit to 46.00 MB. See the query profile for more information about the per-node memory requirements.

The tests were:

00:44:12 [gw3] FAILED query_test/test_queries.py::TestQueries::test_analytic_fns[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:44:21 query_test/test_queries.py::TestQueries::test_limit[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:44:21 [gw3] FAILED query_test/test_queries.py::TestQueries::test_limit[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:44:21 query_test/test_queries.py::TestQueries::test_top_n[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:44:21 [gw3] FAILED query_test/test_queries.py::TestQueries::test_top_n[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:44:25 query_test/test_queries.py::TestQueries::test_union[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 

00:44:32 query_test/test_sort.py::TestQueryFullSort::test_sort_reservation_usage[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:44:32 [gw3] FAILED query_test/test_queries.py::TestQueries::test_union[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:44:35 query_test/test_queries.py::TestQueries::test_sort[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 

00:44:58 [gw3] FAILED query_test/test_queries.py::TestQueries::test_sort[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:44:59 query_test/test_queries.py::TestQueries::test_inline_view[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:44:59 [gw3] FAILED query_test/test_queries.py::TestQueries::test_inline_view[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:45:15 query_test/test_queries.py::TestQueries::test_inline_view_limit[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:45:15 [gw3] FAILED query_test/test_queries.py::TestQueries::test_inline_view_limit[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:47:24 query_test/test_queries.py::TestQueries::test_subquery[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 

00:48:36 [gw3] FAILED query_test/test_queries.py::TestQueries::test_subquery[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:49:01 query_test/test_queries.py::TestQueries::test_empty[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 

00:50:02 [gw3] FAILED query_test/test_queries.py::TestQueries::test_views[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:51:07 query_test/test_queries.py::TestQueries::test_with_clause[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
00:51:12 [gw3] FAILED query_test/test_queries.py::TestQueries::test_with_clause[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] Traceback (most recent call last):

Issue Links

• is broken by: IMPALA-5589 set; shows invalid values for unset query options
• is duplicated by: IMPALA-6017 Multiple tests fail on ASAN due to Rejected query from pool default-pool: minimum memory reservation is greater than memory available
• is duplicated by: IMPALA-5919 Tests exceed internally-set buffer pool memory limit and fail