[IMPALA-9159] test_concurrent_ddls.py's test_ddls_with_invalidate_metadata_sync_ddl timed out after 17 hours

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.4.0
• Fix Version/s: Impala 3.4.0
• Component/s: Frontend
• Labels: broken-build
• Epic Color: ghx-label-14
• Assignee: Quanlong Huang
• Reporter: Joe McDonnell
• Created: 15/Nov/19 22:25
• Updated: 08/Feb/20 00:01
• Resolved: 08/Feb/20 00:01

Description

One run of test_concurrent_ddls.TestConcurrentDdls.test_ddls_with_invalidate_metadata_sync_ddl on erasure coding got stuck for 17 hours:

20:03:55 custom_cluster/test_concurrent_ddls.py::TestConcurrentDdls::test_ddls_with_invalidate_metadata PASSED
13:18:50 custom_cluster/test_concurrent_ddls.py::TestConcurrentDdls::test_ddls_with_invalidate_metadata_sync_ddl 
13:18:50 
13:18:50 **** Tests TIMED OUT! ****

I will attach some of the logs produced. The failure just shows Impala getting killed because the test timed out:

conftest.py:337: in cleanup
    request.instance.execute_query_expect_success(request.instance.client, "use default")
common/impala_test_suite.py:801: in wrapper
    return function(*args, **kwargs)
common/impala_test_suite.py:809: in execute_query_expect_success
    result = cls.__execute_query(impalad_client, query, query_options, user)
common/impala_test_suite.py:904: in __execute_query
    return impalad_client.execute(query, user=user)
common/impala_connection.py:205: in execute
    return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:187: in execute
    handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:362: in __execute_query
    handle = self.execute_query_async(query_string, user=user)
beeswax/impala_beeswax.py:356: in execute_query_async
    handle = self.__do_rpc(lambda: self.imp_service.query(query,))
beeswax/impala_beeswax.py:522: in __do_rpc
    raise ImpalaBeeswaxException(self.__build_error_message(e), e)
E   ImpalaBeeswaxException: ImpalaBeeswaxException:
E    INNER EXCEPTION: <class 'thrift.transport.TTransport.TTransportException'>
E    MESSAGE: TSocket read 0 bytes

As far as I can tell, this has been seen once and that occurrence was on the erasure coding configuration.