[IMPALA-8251] Impala fails to start for TestExchangeDeferredBatches.test_exchange_small_buffer() under release build

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 3.2.0
• Fix Version/s: Impala 3.2.0
• Component/s: Backend
• Labels: broken-build
• Epic Color: ghx-label-1
• Assignee: Michael Ho
• Reporter: Joe McDonnell
• Created: 26/Feb/19 23:31
• Updated: 27/Feb/19 04:13
• Resolved: 27/Feb/19 04:13

Description

custom_cluster/test_exchange_deferred_batches.py is hitting the following error on release builds:

ERROR at setup of TestExchangeDeferredBatches.test_exchange_small_buffer[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
common/custom_cluster_test_suite.py:155: in setup_method
    expected_num_executors=cluster_size)
common/custom_cluster_test_suite.py:230: in _start_impala_cluster
    check_call(cmd + options, close_fds=True)
/usr/lib/python2.7/subprocess.py:541: in check_call
    raise CalledProcessError(retcode, cmd)
E CalledProcessError: Command '['/home/joe/view2/Impala/bin/start-impala-cluster.py', '--cluster_size=3', '--num_coordinators=3', '--log_dir=/tmp/', '--log_level=1', '--impalad_args=--stress_datastream_recvr_delay_ms=3000 --exchg_node_buffer_size_bytes=1024 --datastream_service_num_deserialization_threads=1 ', '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50 ', '--impalad_args=--default_query_options=']' returned non-zero exit status 1

The impalad logs show errors:

Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0226 08:11:15.563117 357 impalad-main.cc:72] Could not find IPv4 address for: jhzvlthd
. Impalad exiting.

It reproduces consistently on my local desktop.

Issue Links

• is broken by: IMPALA-8239 Check failed: deferred_rpcs_.empty() || (num_deserialize_tasks_pending_ + num_pending_enqueue_) > 0