[IMPALA-8115] some jenkins workers slow to do real work due to dpkg lock conflicts

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.4.0
• Component/s: Infrastructure
• Epic Color: ghx-label-4
• Assignee: Laszlo Gaal
• Reporter: Michael Brown
• Created: 24/Jan/19 23:42
• Updated: 21/Feb/20 10:15
• Resolved: 21/Feb/20 10:15

Description

A Jenkins worker for label ubuntu-16.04 took about 15 minutes to start doing real work. I noticed that it was retrying apt-get update:

++ sudo apt-get --yes install openjdk-8-jdk
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
++ date
Thu Jan 24 23:37:33 UTC 2019
++ sudo apt-get update
++ sleep 10
++ sudo apt-get --yes install openjdk-8-jdk
[etc]

I ssh'd into a host and saw that, yes, something else was holding onto the dpkg log (confirmed with lsof and not pasted here. dpkg process PID 11459 was the culprit)

root       1750  0.0  0.0   4508  1664 ?        Ss   23:21   0:00 /bin/sh /usr/lib/apt/apt.systemd.daily
root       1804 12.3  0.1 141076 80452 ?        S    23:22   1:24  \_ /usr/bin/python3 /usr/bin/unattended-upgrade
root       3263  0.0  0.1 140960 72896 ?        S    23:23   0:00      \_ /usr/bin/python3 /usr/bin/unattended-upgrade
root      11459  0.6  0.0  45920 25184 pts/1    Ss+  23:24   0:03          \_ /usr/bin/dpkg --status-fd 10 --unpack --auto-deconfigure /var/cache/apt/archives/tzdata_2018i-0ubuntu0.16.04_all.deb /var/cache/apt/archives/distro-info-data_0.28ubuntu0.9_all.deb /var/cache/apt/archives/file_1%3a5.25-2ubuntu1.1_amd64.deb /var/cache/apt/archives/libmagic1_1%3a5.25-2ubuntu1.1_amd64.deb /var/cache/apt/archives/libisc-export160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libdns-export162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/isc-dhcp-client_4.3.3-5ubuntu12.9_amd64.deb /var/cache/apt/archives/isc-dhcp-common_4.3.3-5ubuntu12.9_amd64.deb /var/cache/apt/archives/libidn11_1.32-3ubuntu1.2_amd64.deb /var/cache/apt/archives/libpng12-0_1.2.54-1ubuntu1.1_amd64.deb /var/cache/apt/archives/libtasn1-6_4.7-3ubuntu0.16.04.3_amd64.deb /var/cache/apt/archives/libapparmor-perl_2.10.95-0ubuntu2.10_amd64.deb /var/cache/apt/archives/apparmor_2.10.95-0ubuntu2.10_amd64.deb /var/cache/apt/archives/curl_7.47.0-1ubuntu2.11_amd64.deb /var/cache/apt/archives/libgssapi-krb5-2_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libkrb5-3_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libkrb5support0_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libk5crypto3_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libcurl3-gnutls_7.47.0-1ubuntu2.11_amd64.deb /var/cache/apt/archives/apt-transport-https_1.2.29ubuntu0.1_amd64.deb /var/cache/apt/archives/libicu55_55.1-7ubuntu0.4_amd64.deb /var/cache/apt/archives/libxml2_2.9.3+dfsg1-1ubuntu0.6_amd64.deb /var/cache/apt/archives/bind9-host_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/dnsutils_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libisc160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libdns162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libisccc140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libisccfg140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/liblwres141_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libbind9-140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/openssl_1.0.2g-1ubuntu4.14_amd64.deb /var/cache/apt/archives/ca-certificates_20170717~16.04.1_all.deb /var/cache/apt/archives/libasprintf0v5_0.19.7-2ubuntu3.1_amd64.deb /var/cache/apt/archives/gettext-base_0.19.7-2ubuntu3.1_amd64.deb /var/cache/apt/archives/krb5-locales_1.13.2+dfsg-5ubuntu2.1_all.deb /var/cache/apt/archives/libelf1_0.165-3ubuntu1.1_amd64.deb /var/cache/apt/archives/libglib2.0-data_2.48.2-0ubuntu4.1_all.deb /var/cache/apt/archives/libnuma1_2.0.11-1ubuntu1.1_amd64.deb /var/cache/apt/archives/libpolkit-gobject-1-0_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/libx11-data_2%3a1.6.3-1ubuntu2.1_all.deb /var/cache/apt/archives/libx11-6_2%3a1.6.3-1ubuntu2.1_amd64.deb /var/cache/apt/archives/openssh-sftp-server_1%3a7.2p2-4ubuntu2.6_amd64.deb /var/cache/apt/archives/openssh-server_1%3a7.2p2-4ubuntu2.6_amd64.deb /var/cache/apt/archives/openssh-client_1%3a7.2p2-4ubuntu2.6_amd64.deb /var/cache/apt/archives/rsync_3.1.1-3ubuntu1.2_amd64.deb /var/cache/apt/archives/tcpdump_4.9.2-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/wget_1.17.1-1ubuntu1.4_amd64.deb /var/cache/apt/archives/python3-problem-report_2.20.1-0ubuntu2.18_all.deb /var/cache/apt/archives/python3-apport_2.20.1-0ubuntu2.18_all.deb /var/cache/apt/archives/apport_2.20.1-0ubuntu2.18_all.deb /var/cache/apt/archives/dns-root-data_2018013001~16.04.1_all.deb /var/cache/apt/archives/dnsmasq-base_2.75-1ubuntu0.16.04.5_amd64.deb /var/cache/apt/archives/git-man_1%3a2.7.4-0ubuntu1.6_all.deb /var/cache/apt/archives/git_1%3a2.7.4-0ubuntu1.6_amd64.deb /var/cache/apt/archives/libmspack0_0.5-1ubuntu0.16.04.3_amd64.deb /var/cache/apt/archives/libpolkit-agent-1-0_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/libpolkit-backend-1-0_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/linux-image-4.4.0-1074-aws_4.4.0-1074.84_amd64.deb /var/cache/apt/archives/linux-aws_4.4.0.1074.76_amd64.deb /var/cache/apt/archives/linux-image-aws_4.4.0.1074.76_amd64.deb /var/cache/apt/archives/linux-aws-headers-4.4.0-1074_4.4.0-1074.84_all.deb /var/cache/apt/archives/linux-headers-4.4.0-1074-aws_4.4.0-1074.84_amd64.deb /var/cache/apt/archives/linux-headers-aws_4.4.0.1074.76_amd64.deb /var/cache/apt/archives/policykit-1_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/python3-cryptography_1.2.3-1ubuntu0.2_amd64.deb /var/cache/apt/archives/python3-jwt_1.3.0-1ubuntu0.1_all.deb /var/cache/apt/archives/python3-requests_2.9.1-3ubuntu0.1_all.deb /var/cache/apt/archives/unattended-upgrades_0.90ubuntu0.10_all.deb
root      12555  0.0  0.0   4500   740 pts/1    S+   23:24   0:00              \_ /bin/sh /var/lib/dpkg/info/unattended-upgrades.prerm upgrade 0.90ubuntu0.10
root      12556  0.0  0.0   4500  1736 pts/1    S+   23:24   0:00                  \_ /bin/sh /usr/sbin/invoke-rc.d unattended-upgrades stop
root      12594  0.0  0.0  24884  1388 pts/1    S+   23:24   0:00                      \_ systemctl stop unattended-upgrades.service

After about 15 minutes, the retry to install openjdk-8-jdk succeeded.

I'm not sure how to make this better except rebuild AMIs with updated packaging. Over time, the system's automatic apt-get update / upgrades are going to take longer and longer and our own stuff is going to block for longer and longer.

Issue Links

• is related to: IMPALA-9319 Automate keeping build images up to date on jenkins.impala.io
• relates to: IMPALA-8767 Jenkins failures due to "Could not get lock /var/lib/dpkg/lock"