Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8820

start-impala-cluster can't find catalogd on recent Ubuntu 16.04

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 3.3.0
    • Impala 3.3.0
    • Infrastructure
    • None

    Description

      We noticed a bug, first on jenkins.impala.io and then on local ubuntu 16.04 environments where start-impala-cluster fails to find a catalogd process, despite the catalogd starting up fine.

      It appears that this is somehow related to an automatic update - it started reproducing for me after I installed the batch of packages on 2019-07-31. I've included the previous upgrade to illustrate the window in which this happened.

      Start-Date: 2019-07-27  00:21:00
      Commandline: apt-get dist-upgrade
      Requested-By: tarmstrong (1000)
      Install: linux-image-4.4.0-157-generic:amd64 (4.4.0-157.185, automatic), linux-headers-4.4.0-157:amd64 (4.4.0-157.185, automatic), linux-tools-4.4.0-157:amd64 (4.4.0-157.185, automatic), linux-tools-4.4.0-157-generic:amd64 (4.4.0-157.185, automatic), linux-headers-4.4.0-157-generic:amd64 (4.4.0-157.185, automatic), linux-modules-extra-4.4.0-157-generic:amd64 (4.4.0-157.185, automatic), linux-modules-4.4.0-157-generic:amd64 (4.4.0-157.185, automatic)
      Upgrade: linux-tools-generic:amd64 (4.4.0.154.162, 4.4.0.157.165), linux-headers-generic:amd64 (4.4.0.154.162, 4.4.0.157.165), linux-libc-dev:amd64 (4.4.0-154.181, 4.4.0-157.185), mysql-client-5.7:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), mysql-server-5.7:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), libmysqlclient-dev:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), linux-image-generic:amd64 (4.4.0.154.162, 4.4.0.157.165), mysql-server:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), mysql-client:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), mysql-client-core-5.7:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), mysql-common:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), libmysqlclient20:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), firefox:amd64 (68.0+build3-0ubuntu0.16.04.1, 68.0.1+build1-0ubuntu0.16.04.1), linux-tools-common:amd64 (4.4.0-154.181, 4.4.0-157.185), patch:amd64 (2.7.5-1ubuntu0.16.04.1, 2.7.5-1ubuntu0.16.04.2), linux-generic:amd64 (4.4.0.154.162, 4.4.0.157.165), mysql-server-core-5.7:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), docker-ce:amd64 (5:19.03.0~3-0~ubuntu-xenial, 5:19.03.1~3-0~ubuntu-xenial), docker-ce-cli:amd64 (5:19.03.0~3-0~ubuntu-xenial, 5:19.03.1~3-0~ubuntu-xenial)
      End-Date: 2019-07-27  00:22:33
      
      Start-Date: 2019-07-31  12:55:00
      Commandline: apt-get dist-upgrade
      Requested-By: tarmstrong (1000)
      Upgrade: openjdk-8-jdk:amd64 (8u212-b03-0ubuntu1.16.04.1, 8u222-b10-1ubuntu1~16.04.1), libldap-2.4-2:amd64 (2.4.42+dfsg-2ubuntu3.5, 2.4.42+dfsg-2ubuntu3.6), openjdk-8-jre:amd64 (8u212-b03-0ubuntu1.16.04.1, 8u222-b10-1ubuntu1~16.04.1), slack-desktop:amd64 (4.0.0, 4.0.1), libsvn1:amd64 (1.9.3-2ubuntu1.1, 1.9.3-2ubuntu1.3), openjdk-8-jdk-headless:amd64 (8u212-b03-0ubuntu1.16.04.1, 8u222-b10-1ubuntu1~16.04.1), libsvn-perl:amd64 (1.9.3-2ubuntu1.1, 1.9.3-2ubuntu1.3), subversion:amd64 (1.9.3-2ubuntu1.1, 1.9.3-2ubuntu1.3), openjdk-8-jre-headless:amd64 (8u212-b03-0ubuntu1.16.04.1, 8u222-b10-1ubuntu1~16.04.1)
      End-Date: 2019-07-31  12:55:14
      

      This issue is that the catalogd process's name is now "main", instead of "catalogd".

      I think we can fix by changing our scripts to fall back to checking the binary name.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tarmstrong Tim Armstrong
            tarmstrong Tim Armstrong
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment