Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4839

Kudu-related tests failing on remote cluster because localhost / loopback is hard-coded in the test framework

    Details

      Description

      There are a couple of places that localhost and/or the loopback address is hardcoded in the Impala test framework. This is causing tests in query_test/test_kudu.py to fail when run on a remote cluster.

      The first place is in conftest.py, where the kudu_client fixture makes use of the impala.dbapi.connect() method from the impyla library. Because host param is not being passed when the method is called, it uses a default of localhost, resulting in the following error:

      conftest.py:339: in conn
          with __unique_conn(db_name=db_name, timeout=timeout) as conn:
      /usr/lib/python2.7/contextlib.py:17: in __enter__
          return self.gen.next()
      conftest.py:370: in __unique_conn
          with __auto_closed_conn() as conn:
      /usr/lib/python2.7/contextlib.py:17: in __enter__
          return self.gen.next()
      conftest.py:395: in __auto_closed_conn
          conn = impala_connect(database=db_name, timeout=timeout)
      ../infra/python/env/local/lib/python2.7/site-packages/impala/dbapi.py:147: in connect
          auth_mechanism=auth_mechanism)
      ../infra/python/env/local/lib/python2.7/site-packages/impala/hiveserver2.py:758: in connect
          transport.open()
      ../infra/python/env/local/lib/python2.7/site-packages/thrift/transport/TTransport.py:149: in open
          return self.__trans.open()
      ../infra/python/env/local/lib/python2.7/site-packages/thrift/transport/TSocket.py:99: in open
          message=message)
      E   TTransportException: Could not connect to localhost:21050
      

      After correcting for that, the next problem is that KUDU_MASTER_HOSTS is currently defined in common/_init_.py as 127.0.0.1, and there's no way to override it with a pytest param:

      $ cat common/__init__.py
      # These values should match the impalad startup flag -kudu_master_hosts
      KUDU_MASTER_HOSTS = "127.0.0.1"
      

      As a result, the following error is thrown:

      conftest.py:310: in kudu_client
          kudu_client = kudu_connect(host, port)
      ../infra/python/env/local/lib/python2.7/site-packages/kudu/__init__.py:92: in connect
          rpc_timeout_ms=rpc_timeout_ms)
      kudu/client.pyx:271: in kudu.client.Client.__cinit__ (kudu/client.cpp:5298)
          ???
      kudu/errors.pyx:62: in kudu.errors.check_status (kudu/errors.cpp:1074)
          ???
      E   KuduBadStatus: Timed out: Could not locate the leader master: GetMasterRegistration RPC to 127.0.0.1:7051 timed out after 0.227s
      

        Activity

        Hide
        lv Lars Volker added a comment -

        I'm changing the "Fix Version/s" to 2.9 as part of a bulk edit. If you think this is wrong, please set the version to the correct one.

        Show
        lv Lars Volker added a comment - I'm changing the "Fix Version/s" to 2.9 as part of a bulk edit. If you think this is wrong, please set the version to the correct one.
        Hide
        dknupp David Knupp added a comment -

        Parent: 9e36088c (IMPALA-4920: custom cluster tests: fix generation of py.test options)
        Author: David Knupp <dknupp@cloudera.com>
        AuthorDate: 2017-02-01 16:03:28 -0800
        Commit: Impala Public Jenkins <impala-public-jenkins@gerrit.cloudera.org>
        CommitDate: 2017-02-14 21:51:39 +0000

        IMPALA-4839: Remove implicit 'localhost' for KUDU_MASTER_HOSTS

        The Kudu query tests were failing on a remote cluster because the Kudu
        master was always set to '127.0.0.1', with no way to override it.

        This patch corrects the issue with a number of changes:

        • Add a pytest command line option to specify an arbitrary Kudu master
        • Consolidate the place where the default Kudu master is derived. It
          had been stored both in the env and in tests/common/_init_.py,
          with different files looking to different places. For now, just look
          to the env, and remove the value from _init_.py.
        • The kudu_client test fixture in conftest.py was using the connect()
          method from impala.dbapi (part of the Impyla library), without
          specifying the host param. In the absence of that, the default value
          is 'localhost', so add the host param to the connect() call.
        • Define the various defaults for pytest config as constants at the top
          of conftest.py.

        Change-Id: I9df71480a165f4ce21ae3edab6ce7227fbf76f77
        Reviewed-on: http://gerrit.cloudera.org:8080/5877
        Reviewed-by: Matthew Jacobs <mj@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        dknupp David Knupp added a comment - Parent: 9e36088c ( IMPALA-4920 : custom cluster tests: fix generation of py.test options) Author: David Knupp <dknupp@cloudera.com> AuthorDate: 2017-02-01 16:03:28 -0800 Commit: Impala Public Jenkins <impala-public-jenkins@gerrit.cloudera.org> CommitDate: 2017-02-14 21:51:39 +0000 IMPALA-4839 : Remove implicit 'localhost' for KUDU_MASTER_HOSTS The Kudu query tests were failing on a remote cluster because the Kudu master was always set to '127.0.0.1', with no way to override it. This patch corrects the issue with a number of changes: Add a pytest command line option to specify an arbitrary Kudu master Consolidate the place where the default Kudu master is derived. It had been stored both in the env and in tests/common/_ init _.py, with different files looking to different places. For now, just look to the env, and remove the value from _ init _.py. The kudu_client test fixture in conftest.py was using the connect() method from impala.dbapi (part of the Impyla library), without specifying the host param. In the absence of that, the default value is 'localhost', so add the host param to the connect() call. Define the various defaults for pytest config as constants at the top of conftest.py. Change-Id: I9df71480a165f4ce21ae3edab6ce7227fbf76f77 Reviewed-on: http://gerrit.cloudera.org:8080/5877 Reviewed-by: Matthew Jacobs <mj@cloudera.com> Tested-by: Impala Public Jenkins
        Hide
        dknupp David Knupp added a comment -
        Show
        dknupp David Knupp added a comment - Up for review: https://gerrit.cloudera.org/#/c/5877/
        Hide
        dknupp David Knupp added a comment -

        Michael Brown Noted. Thanks.

        Show
        dknupp David Knupp added a comment - Michael Brown Noted. Thanks.
        Hide
        mikesbrown Michael Brown added a comment -

        I'm not sure two places is wrong in the case where someone did the right thing (fixtures). If anything, I'd say other clients should be other fixtures, no?

        The history of that code starts here: https://gerrit.cloudera.org/#/c/2855/

        Show
        mikesbrown Michael Brown added a comment - I'm not sure two places is wrong in the case where someone did the right thing (fixtures). If anything, I'd say other clients should be other fixtures, no? The history of that code starts here: https://gerrit.cloudera.org/#/c/2855/
        Hide
        dknupp David Knupp added a comment -

        Michael Brown KUDU_MASTER_HOSTS is primarily consumed in two places:

        • query_test/test_kudu.py, not surprisingly
        • conftest.py to set up the kudu_client fixture

        It's a little weird the way that the kudu_client is defined. It's the only client that's defined as a test fixture. A cursory glance shows that other clients (hdfs, impala, s3, etc.) all get defined in ImpalaTestSuite. I'm not saying that one is better than the other, but having two places is bad.

        Do you have any immediate ideas on how to resolve this discrepancy. A quick compromise might be to move the kudu_client to the KuduTestSuite, but I'm open to other suggestions.

        (I guess my gut feeling is that there should be something like a central common/clients.py module from which all of the clients could be imported as needed, maybe? But that would be a large change.)

        Show
        dknupp David Knupp added a comment - Michael Brown KUDU_MASTER_HOSTS is primarily consumed in two places: query_test/test_kudu.py , not surprisingly conftest.py to set up the kudu_client fixture It's a little weird the way that the kudu_client is defined. It's the only client that's defined as a test fixture. A cursory glance shows that other clients (hdfs, impala, s3, etc.) all get defined in ImpalaTestSuite. I'm not saying that one is better than the other, but having two places is bad. Do you have any immediate ideas on how to resolve this discrepancy. A quick compromise might be to move the kudu_client to the KuduTestSuite , but I'm open to other suggestions. (I guess my gut feeling is that there should be something like a central common/clients.py module from which all of the clients could be imported as needed, maybe? But that would be a large change.)

          People

          • Assignee:
            dknupp David Knupp
            Reporter:
            dknupp David Knupp
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development