[IMPALA-8820] start-impala-cluster can't find catalogd on recent Ubuntu 16.04

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 3.3.0
• Fix Version/s: Impala 3.3.0
• Component/s: Infrastructure
• Target Version: Impala 3.3.0
• Epic Color: ghx-label-7
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 31/Jul/19 20:22
• Updated: 01/Aug/19 15:27
• Resolved: 01/Aug/19 15:27

Description

We noticed a bug, first on jenkins.impala.io and then on local ubuntu 16.04 environments where start-impala-cluster fails to find a catalogd process, despite the catalogd starting up fine.

It appears that this is somehow related to an automatic update - it started reproducing for me after I installed the batch of packages on 2019-07-31. I've included the previous upgrade to illustrate the window in which this happened.

Start-Date: 2019-07-27  00:21:00
Commandline: apt-get dist-upgrade
Requested-By: tarmstrong (1000)
Install: linux-image-4.4.0-157-generic:amd64 (4.4.0-157.185, automatic), linux-headers-4.4.0-157:amd64 (4.4.0-157.185, automatic), linux-tools-4.4.0-157:amd64 (4.4.0-157.185, automatic), linux-tools-4.4.0-157-generic:amd64 (4.4.0-157.185, automatic), linux-headers-4.4.0-157-generic:amd64 (4.4.0-157.185, automatic), linux-modules-extra-4.4.0-157-generic:amd64 (4.4.0-157.185, automatic), linux-modules-4.4.0-157-generic:amd64 (4.4.0-157.185, automatic)
Upgrade: linux-tools-generic:amd64 (4.4.0.154.162, 4.4.0.157.165), linux-headers-generic:amd64 (4.4.0.154.162, 4.4.0.157.165), linux-libc-dev:amd64 (4.4.0-154.181, 4.4.0-157.185), mysql-client-5.7:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), mysql-server-5.7:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), libmysqlclient-dev:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), linux-image-generic:amd64 (4.4.0.154.162, 4.4.0.157.165), mysql-server:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), mysql-client:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), mysql-client-core-5.7:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), mysql-common:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), libmysqlclient20:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), firefox:amd64 (68.0+build3-0ubuntu0.16.04.1, 68.0.1+build1-0ubuntu0.16.04.1), linux-tools-common:amd64 (4.4.0-154.181, 4.4.0-157.185), patch:amd64 (2.7.5-1ubuntu0.16.04.1, 2.7.5-1ubuntu0.16.04.2), linux-generic:amd64 (4.4.0.154.162, 4.4.0.157.165), mysql-server-core-5.7:amd64 (5.7.26-0ubuntu0.16.04.1, 5.7.27-0ubuntu0.16.04.1), docker-ce:amd64 (5:19.03.0~3-0~ubuntu-xenial, 5:19.03.1~3-0~ubuntu-xenial), docker-ce-cli:amd64 (5:19.03.0~3-0~ubuntu-xenial, 5:19.03.1~3-0~ubuntu-xenial)
End-Date: 2019-07-27  00:22:33

Start-Date: 2019-07-31  12:55:00
Commandline: apt-get dist-upgrade
Requested-By: tarmstrong (1000)
Upgrade: openjdk-8-jdk:amd64 (8u212-b03-0ubuntu1.16.04.1, 8u222-b10-1ubuntu1~16.04.1), libldap-2.4-2:amd64 (2.4.42+dfsg-2ubuntu3.5, 2.4.42+dfsg-2ubuntu3.6), openjdk-8-jre:amd64 (8u212-b03-0ubuntu1.16.04.1, 8u222-b10-1ubuntu1~16.04.1), slack-desktop:amd64 (4.0.0, 4.0.1), libsvn1:amd64 (1.9.3-2ubuntu1.1, 1.9.3-2ubuntu1.3), openjdk-8-jdk-headless:amd64 (8u212-b03-0ubuntu1.16.04.1, 8u222-b10-1ubuntu1~16.04.1), libsvn-perl:amd64 (1.9.3-2ubuntu1.1, 1.9.3-2ubuntu1.3), subversion:amd64 (1.9.3-2ubuntu1.1, 1.9.3-2ubuntu1.3), openjdk-8-jre-headless:amd64 (8u212-b03-0ubuntu1.16.04.1, 8u222-b10-1ubuntu1~16.04.1)
End-Date: 2019-07-31  12:55:14

This issue is that the catalogd process's name is now "main", instead of "catalogd".

I think we can fix by changing our scripts to fall back to checking the binary name.