[IMPALA-10562] TestGracefulShutdown::test_shutdown_idle failes in GCE instance

• Type: Test
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Epic Color: ghx-label-5
• Assignee: Unassigned
• Reporter: Quanlong Huang
• Created: 08/Mar/21 08:13
• Updated: 13/Mar/21 13:01

Description

Google Cloud will add a line in /etc/hosts for resolving the hostname of the instance. E.g.

[systest@quanlong-compile impala]$ cat /etc/hosts                                                                                                                                                                                    
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
x.x.x.x quanlong-compile.gce.cloudera.com quanlong-compile  # Added by Google
x.x.x.x metadata.google.internal  # Added by Google

127.0.0.1 quanlong-compile quanlong-compile.gce.cloudera.com

(The IPs are manually masked)

This causes connection refused failure in TestGracefulShutdown::test_shutdown_idle

tests/custom_cluster/test_restart_services.py:295: in test_shutdown_idle
    query_options={})
tests/common/impala_test_suite.py:814: in wrapper
    return function(*args, **kwargs)
tests/common/impala_test_suite.py:822: in execute_query_expect_success
    result = cls.__execute_query(impalad_client, query, query_options, user)
tests/common/impala_test_suite.py:923: in __execute_query
    return impalad_client.execute(query, user=user)
tests/common/impala_connection.py:205: in execute
    return self.__beeswax_client.execute(sql_stmt, user=user)
tests/beeswax/impala_beeswax.py:187: in execute
    handle = self.__execute_query(query_string.strip(), user=user)
tests/beeswax/impala_beeswax.py:363: in __execute_query
    handle = self.execute_query_async(query_string, user=user)
tests/beeswax/impala_beeswax.py:357: in execute_query_async
    handle = self.__do_rpc(lambda: self.imp_service.query(query,))
tests/beeswax/impala_beeswax.py:520: in __do_rpc
    raise ImpalaBeeswaxException(self.__build_error_message(b), b)
E   ImpalaBeeswaxException: ImpalaBeeswaxException:
E    INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
E    MESSAGE: Rpc to 127.0.0.1:27001 failed with error 'RemoteShutdown() RPC failed: Network error: Client connection negotiation failed: client connection to 127.0.0.1:27001: connect: Connection refused (error 111)'

The failed command is ":shutdown('localhost:27001')".

Checked with the impalad, it's listening on "quanlong-compile.gce.cloudera.com:27001":

[systest@quanlong-compile impala]$ ps aux | grep 28771
systest  28771 11.0  3.5 22579424 2165200 pts/1 Sl  23:28   0:15 /home/systest/impala/be/build/latest/service/impalad -disconnected_session_timeout 21600 -kudu_client_rpc_timeout_ms 0 -kudu_master_hosts localhost -mem_limit=12884
901888 -logbufsecs=5 -v=1 -max_log_files=10 -log_filename=impalad_node1 -log_dir=/tmp/ -beeswax_port=21001 -hs2_port=21051 -hs2_http_port=28001 -krpc_port=27001 -state_store_subscriber_port=23001 -webserver_port=25001 --shutdown_
grace_period_s=1 --hostname=quanlong-compile.gce.cloudera.com --default_query_options=
systest  30727  0.0  0.0 112828  1032 pts/1    S+   23:31   0:00 grep --color=auto 28771
[systest@quanlong-compile impala]$ lsof -p 28771 | grep 27001
impalad 28771 systest  341u     IPv4             577338       0t0       TCP quanlong-compile.gce.cloudera.com:27001 (LISTEN)

The shutdown command works with hostname:

[localhost.localdomain:21050] default> :shutdown('localhost:27001');
Query: :shutdown('localhost:27001')
Query submitted at: 2021-03-07 23:39:25 (Coordinator: http://quanlong-compile.gce.cloudera.com:25000)
ERROR: Rpc to 127.0.0.1:27001 failed with error 'RemoteShutdown() RPC failed: Network error: Client connection negotiation failed: client connection to 127.0.0.1:27001: connect: Connection refused (error 111)'

[localhost.localdomain:21050] default> :shutdown('quanlong-compile.gce.cloudera.com:27001');                                                                                                                                         
Query: :shutdown('quanlong-compile.gce.cloudera.com:27001')
Query submitted at: 2021-03-07 23:41:35 (Coordinator: http://quanlong-compile.gce.cloudera.com:25000)
Query progress can be monitored at: http://quanlong-compile.gce.cloudera.com:25000/query_plan?query_id=81475574dca0db03:6e2f767d00000000
+-------------------------------------------------------------------------------------------------------------------------------------------+
| summary                                                                                                                                   |
+-------------------------------------------------------------------------------------------------------------------------------------------+
| shutdown grace period left: 1s000ms, deadline left: 1h, queries registered on coordinator: 0, queries executing: 0, fragment instances: 0 |
+-------------------------------------------------------------------------------------------------------------------------------------------+
Fetched 1 row(s) in 0.01s

Removing the line "x.x.x.x quanlong-compile.gce.cloudera.com quanlong-compile # Added by Google" can pass the test. Maybe we should add the logics in bin/bootstrap_system.sh when running in GCE instance.

Issue Links

• relates to: IMPALA-7712 Impala read from and write to GCS