[IMPALA-7013] test_insert_permutation failed with "timed out waiting for receiver fragment instance"

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.13.0
• Component/s: Distributed Exec
• Labels: flaky
• Target Version: Impala 2.13.0
• Epic Color: ghx-label-9
• Assignee: Joe McDonnell
• Reporter: Tim Armstrong
• Created: 10/May/18 22:46
• Updated: 04/Oct/18 17:56
• Resolved: 04/Oct/18 17:56

Description

This is a weird failure that I don't quite understand

11:32:57  TestInsertQueriesWithPermutation.test_insert_permutation[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none] 
11:32:57 [gw2] linux2 -- Python 2.6.6 /data/jenkins/workspace/impala-cdh5-trunk-core-s3/repos/Impala/bin/../infra/python/env/bin/python
11:32:57 query_test/test_insert_permutation.py:50: in test_insert_permutation
11:32:57     self.run_test_case('QueryTest/insert_permutation', vector)
11:32:57 common/impala_test_suite.py:405: in run_test_case
11:32:57     result = self.__execute_query(target_impalad_client, query, user=user)
11:32:57 common/impala_test_suite.py:620: in __execute_query
11:32:57     return impalad_client.execute(query, user=user)
11:32:57 common/impala_connection.py:160: in execute
11:32:57     return self.__beeswax_client.execute(sql_stmt, user=user)
11:32:57 beeswax/impala_beeswax.py:173: in execute
11:32:57     handle = self.__execute_query(query_string.strip(), user=user)
11:32:57 beeswax/impala_beeswax.py:341: in __execute_query
11:32:57     self.wait_for_completion(handle)
11:32:57 beeswax/impala_beeswax.py:361: in wait_for_completion
11:32:57     raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
11:32:57 E   ImpalaBeeswaxException: ImpalaBeeswaxException:
11:32:57 E    Query aborted:Sender 127.0.0.1 timed out waiting for receiver fragment instance: 994ce8cbd8cb4eb2:f1cb807c00000000, dest node: 1
11:32:57 ---------------------------- Captured stderr setup -----------------------------
11:32:57 -- connecting to: localhost:21000
11:32:57 ----------------------------- Captured stderr call -----------------------------
11:32:57 -- executing against localhost:21000
11:32:57 use default;
11:32:57 
11:32:57 SET sync_ddl=1;
11:32:57 -- executing against localhost:21000
11:32:57 drop database if exists `insert_permutation_test` cascade;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 use functional;
11:32:57 
11:32:57 SET batch_size=0;
11:32:57 SET num_nodes=0;
11:32:57 SET disable_codegen_rows_threshold=0;
11:32:57 SET disable_codegen=False;
11:32:57 SET abort_on_error=1;
11:32:57 SET exec_single_node_rows_threshold=0;
11:32:57 -- executing against localhost:21000
11:32:57 create database insert_permutation_test location
11:32:57 's3a://impala-cdh5-s3-test/test-warehouse/insert_permutation_test';
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 use insert_permutation_test;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 create table perm_nopart(int_col1 int, string_col string, int_col2 int);
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 
11:32:57 create table perm_part(int_col1 int, string_col string) partitioned by (p1 int, p2 string);
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 
11:32:57 create table parquet_part(int_col1 int, string_col string)
11:32:57 partitioned by (p1 int, p2 string) stored as parquet;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_nopart;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_nopart(int_col1, string_col, int_col2) values(1,'str',2);
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_nopart;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_nopart;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_nopart(int_col2, string_col, int_col1) values(1,'str',2);
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_nopart;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_nopart;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_nopart(int_col2) values(1);
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_nopart;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_part(p1, string_col, int_col1, p2) values(10,'str',1, 'hello');
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_part(p2, string_col, int_col1, p1) values('hello','str',1, 10);
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_part(p2, p1) values('hello', 10);
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_part(p2) PARTITION(p1=10) values('hello');
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_part(int_col1, string_col) PARTITION(p1=10, p2) values(1,'perm_col','part_col');
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_part() PARTITION(p1=10, p2='foo');
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_part() PARTITION(p1, p2='foo') values(5);
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_nopart;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_nopart();
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_nopart;