[IMPALA-4839] Kudu-related tests failing on remote cluster because localhost / loopback is hard-coded in the test framework

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.9.0
• Component/s: Infrastructure
• Labels: kudu remote_cluster_test test-infra
• Assignee: David Knupp
• Reporter: David Knupp
• Created: 30/Jan/17 04:45
• Updated: 16/Jun/17 19:01
• Resolved: 23/Feb/17 16:11

Description

There are a couple of places that localhost and/or the loopback address is hardcoded in the Impala test framework. This is causing tests in query_test/test_kudu.py to fail when run on a remote cluster.

The first place is in conftest.py, where the kudu_client fixture makes use of the impala.dbapi.connect() method from the impyla library. Because host param is not being passed when the method is called, it uses a default of localhost, resulting in the following error:

conftest.py:339: in conn
    with __unique_conn(db_name=db_name, timeout=timeout) as conn:
/usr/lib/python2.7/contextlib.py:17: in __enter__
    return self.gen.next()
conftest.py:370: in __unique_conn
    with __auto_closed_conn() as conn:
/usr/lib/python2.7/contextlib.py:17: in __enter__
    return self.gen.next()
conftest.py:395: in __auto_closed_conn
    conn = impala_connect(database=db_name, timeout=timeout)
../infra/python/env/local/lib/python2.7/site-packages/impala/dbapi.py:147: in connect
    auth_mechanism=auth_mechanism)
../infra/python/env/local/lib/python2.7/site-packages/impala/hiveserver2.py:758: in connect
    transport.open()
../infra/python/env/local/lib/python2.7/site-packages/thrift/transport/TTransport.py:149: in open
    return self.__trans.open()
../infra/python/env/local/lib/python2.7/site-packages/thrift/transport/TSocket.py:99: in open
    message=message)
E   TTransportException: Could not connect to localhost:21050

After correcting for that, the next problem is that KUDU_MASTER_HOSTS is currently defined in common/_init_.py as 127.0.0.1, and there's no way to override it with a pytest param:

$ cat common/__init__.py
# These values should match the impalad startup flag -kudu_master_hosts
KUDU_MASTER_HOSTS = "127.0.0.1"

As a result, the following error is thrown:

conftest.py:310: in kudu_client
    kudu_client = kudu_connect(host, port)
../infra/python/env/local/lib/python2.7/site-packages/kudu/__init__.py:92: in connect
    rpc_timeout_ms=rpc_timeout_ms)
kudu/client.pyx:271: in kudu.client.Client.__cinit__ (kudu/client.cpp:5298)
    ???
kudu/errors.pyx:62: in kudu.errors.check_status (kudu/errors.cpp:1074)
    ???
E   KuduBadStatus: Timed out: Could not locate the leader master: GetMasterRegistration RPC to 127.0.0.1:7051 timed out after 0.227s