Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4684

check-hbase-nodes.py: Build failing on RHEL7 when trying to start HBase

    Details

      Description

      I've seen this once when starting a minicluster:

      Contents of HDFS root: []
      Connecting to Zookeeper host(s).
      No handlers could be found for logger "kazoo.client"
      Success: <kazoo.client.KazooClient object at 0xc5a590>
      Waiting for HBase node: /hbase/master
      Waiting for HBase node: /hbase/master
      Traceback (most recent call last):
        File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/testdata/bin/check-hbase-nodes.py", line 167, in <module>
          errors = check_znodes_list_for_errors(args.nodes, args.zookeeper_hosts, args.timeout)
        File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/testdata/bin/check-hbase-nodes.py", line 135, in check_znodes_list_for_errors
          zk_client.stop()
        File "/usr/lib64/python2.7/contextlib.py", line 154, in __exit__
          self.thing.close()
        File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/infra/python/env/lib/python2.7/site-packages/kazoo/client.py", line 612, in close
          self._connection.close()
        File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/infra/python/env/lib/python2.7/site-packages/kazoo/protocol/connection.py", line 192, in close
          raise Exception("Cannot close connection until it is stopped")
      Exception: Cannot close connection until it is stopped
      

        Activity

        Hide
        mikesbrown Michael Brown added a comment -

        This seems to happen regularity when trying to start a minicluster on RHEL7.

        Show
        mikesbrown Michael Brown added a comment - This seems to happen regularity when trying to start a minicluster on RHEL7.
        Hide
        dknupp David Knupp added a comment -

        I have an idea for a fix. Will try to get it in today.

        Show
        dknupp David Knupp added a comment - I have an idea for a fix. Will try to get it in today.
        Hide
        dknupp David Knupp added a comment -

        Patch submitted for review: https://gerrit.cloudera.org/#/c/5547/

        Show
        dknupp David Knupp added a comment - Patch submitted for review: https://gerrit.cloudera.org/#/c/5547/
        Hide
        dknupp David Knupp added a comment -

        The original description is somewhat misleading. Exception("Cannot close connection until it is stopped") is more of a symptom than a cause. The actual underlying error is a Zookeeper connection loss exception.

        Traceback (most recent call last):
          File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/testdata/bin/check-hbase-nodes.py", line 169, in <module>
            errors = check_znodes_list_for_errors(args.nodes, args.zookeeper_hosts, args.timeout)
          File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/testdata/bin/check-hbase-nodes.py", line 135, in check_znodes_list_for_errors
            errors = sum([check_znode(node, zk_client, timeout) for node in nodes])
          File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/testdata/bin/check-hbase-nodes.py", line 111, in check_znode
            node_info = zk_client.get(node)
          File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/infra/python/env/lib/python2.7/site-packages/kazoo/client.py", line 1026, in get
            return self.get_async(path, watch).get()
          File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/infra/python/env/lib/python2.7/site-packages/kazoo/handlers/utils.py", line 78, in get
            raise self._exception
        kazoo.exceptions.ConnectionLoss
        
        Show
        dknupp David Knupp added a comment - The original description is somewhat misleading. Exception("Cannot close connection until it is stopped") is more of a symptom than a cause. The actual underlying error is a Zookeeper connection loss exception. Traceback (most recent call last): File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/testdata/bin/check-hbase-nodes.py", line 169, in <module> errors = check_znodes_list_for_errors(args.nodes, args.zookeeper_hosts, args.timeout) File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/testdata/bin/check-hbase-nodes.py", line 135, in check_znodes_list_for_errors errors = sum([check_znode(node, zk_client, timeout) for node in nodes]) File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/testdata/bin/check-hbase-nodes.py", line 111, in check_znode node_info = zk_client.get(node) File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/infra/python/env/lib/python2.7/site-packages/kazoo/client.py", line 1026, in get return self.get_async(path, watch).get() File "/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/infra/python/env/lib/python2.7/site-packages/kazoo/handlers/utils.py", line 78, in get raise self._exception kazoo.exceptions.ConnectionLoss
        Hide
        dknupp David Knupp added a comment -

        New patch submitted for review: https://gerrit.cloudera.org/#/c/5554

        Show
        dknupp David Knupp added a comment - New patch submitted for review: https://gerrit.cloudera.org/#/c/5554
        Hide
        dknupp David Knupp added a comment -

        Change has been successfully cherry-picked as 226a2e63321e9bcf4ba906512fc40e35b98db252

        Show
        dknupp David Knupp added a comment - Change has been successfully cherry-picked as 226a2e63321e9bcf4ba906512fc40e35b98db252
        Hide
        tarasbob Taras Bobrovytsky added a comment -

        An issue similar to this one happened again on RHEL7.

        Show
        tarasbob Taras Bobrovytsky added a comment - An issue similar to this one happened again on RHEL7.
        Hide
        tarasbob Taras Bobrovytsky added a comment -

        Reopened IMPALA-4733 instead.

        Show
        tarasbob Taras Bobrovytsky added a comment - Reopened IMPALA-4733 instead.
        Hide
        lv Lars Volker added a comment -

        I'm changing the "Fix Version/s" to 2.9 as part of a bulk edit. If you think this is wrong, please set the version to the correct one.

        Show
        lv Lars Volker added a comment - I'm changing the "Fix Version/s" to 2.9 as part of a bulk edit. If you think this is wrong, please set the version to the correct one.

          People

          • Assignee:
            dknupp David Knupp
            Reporter:
            mikesbrown Michael Brown
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development