Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
ghx-label-4
Description
A Jenkins worker for label ubuntu-16.04 took about 15 minutes to start doing real work. I noticed that it was retrying apt-get update:
++ sudo apt-get --yes install openjdk-8-jdk E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable) E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it? ++ date Thu Jan 24 23:37:33 UTC 2019 ++ sudo apt-get update ++ sleep 10 ++ sudo apt-get --yes install openjdk-8-jdk [etc]
I ssh'd into a host and saw that, yes, something else was holding onto the dpkg log (confirmed with lsof and not pasted here. dpkg process PID 11459 was the culprit)
root 1750 0.0 0.0 4508 1664 ? Ss 23:21 0:00 /bin/sh /usr/lib/apt/apt.systemd.daily root 1804 12.3 0.1 141076 80452 ? S 23:22 1:24 \_ /usr/bin/python3 /usr/bin/unattended-upgrade root 3263 0.0 0.1 140960 72896 ? S 23:23 0:00 \_ /usr/bin/python3 /usr/bin/unattended-upgrade root 11459 0.6 0.0 45920 25184 pts/1 Ss+ 23:24 0:03 \_ /usr/bin/dpkg --status-fd 10 --unpack --auto-deconfigure /var/cache/apt/archives/tzdata_2018i-0ubuntu0.16.04_all.deb /var/cache/apt/archives/distro-info-data_0.28ubuntu0.9_all.deb /var/cache/apt/archives/file_1%3a5.25-2ubuntu1.1_amd64.deb /var/cache/apt/archives/libmagic1_1%3a5.25-2ubuntu1.1_amd64.deb /var/cache/apt/archives/libisc-export160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libdns-export162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/isc-dhcp-client_4.3.3-5ubuntu12.9_amd64.deb /var/cache/apt/archives/isc-dhcp-common_4.3.3-5ubuntu12.9_amd64.deb /var/cache/apt/archives/libidn11_1.32-3ubuntu1.2_amd64.deb /var/cache/apt/archives/libpng12-0_1.2.54-1ubuntu1.1_amd64.deb /var/cache/apt/archives/libtasn1-6_4.7-3ubuntu0.16.04.3_amd64.deb /var/cache/apt/archives/libapparmor-perl_2.10.95-0ubuntu2.10_amd64.deb /var/cache/apt/archives/apparmor_2.10.95-0ubuntu2.10_amd64.deb /var/cache/apt/archives/curl_7.47.0-1ubuntu2.11_amd64.deb /var/cache/apt/archives/libgssapi-krb5-2_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libkrb5-3_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libkrb5support0_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libk5crypto3_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libcurl3-gnutls_7.47.0-1ubuntu2.11_amd64.deb /var/cache/apt/archives/apt-transport-https_1.2.29ubuntu0.1_amd64.deb /var/cache/apt/archives/libicu55_55.1-7ubuntu0.4_amd64.deb /var/cache/apt/archives/libxml2_2.9.3+dfsg1-1ubuntu0.6_amd64.deb /var/cache/apt/archives/bind9-host_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/dnsutils_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libisc160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libdns162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libisccc140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libisccfg140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/liblwres141_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libbind9-140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/openssl_1.0.2g-1ubuntu4.14_amd64.deb /var/cache/apt/archives/ca-certificates_20170717~16.04.1_all.deb /var/cache/apt/archives/libasprintf0v5_0.19.7-2ubuntu3.1_amd64.deb /var/cache/apt/archives/gettext-base_0.19.7-2ubuntu3.1_amd64.deb /var/cache/apt/archives/krb5-locales_1.13.2+dfsg-5ubuntu2.1_all.deb /var/cache/apt/archives/libelf1_0.165-3ubuntu1.1_amd64.deb /var/cache/apt/archives/libglib2.0-data_2.48.2-0ubuntu4.1_all.deb /var/cache/apt/archives/libnuma1_2.0.11-1ubuntu1.1_amd64.deb /var/cache/apt/archives/libpolkit-gobject-1-0_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/libx11-data_2%3a1.6.3-1ubuntu2.1_all.deb /var/cache/apt/archives/libx11-6_2%3a1.6.3-1ubuntu2.1_amd64.deb /var/cache/apt/archives/openssh-sftp-server_1%3a7.2p2-4ubuntu2.6_amd64.deb /var/cache/apt/archives/openssh-server_1%3a7.2p2-4ubuntu2.6_amd64.deb /var/cache/apt/archives/openssh-client_1%3a7.2p2-4ubuntu2.6_amd64.deb /var/cache/apt/archives/rsync_3.1.1-3ubuntu1.2_amd64.deb /var/cache/apt/archives/tcpdump_4.9.2-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/wget_1.17.1-1ubuntu1.4_amd64.deb /var/cache/apt/archives/python3-problem-report_2.20.1-0ubuntu2.18_all.deb /var/cache/apt/archives/python3-apport_2.20.1-0ubuntu2.18_all.deb /var/cache/apt/archives/apport_2.20.1-0ubuntu2.18_all.deb /var/cache/apt/archives/dns-root-data_2018013001~16.04.1_all.deb /var/cache/apt/archives/dnsmasq-base_2.75-1ubuntu0.16.04.5_amd64.deb /var/cache/apt/archives/git-man_1%3a2.7.4-0ubuntu1.6_all.deb /var/cache/apt/archives/git_1%3a2.7.4-0ubuntu1.6_amd64.deb /var/cache/apt/archives/libmspack0_0.5-1ubuntu0.16.04.3_amd64.deb /var/cache/apt/archives/libpolkit-agent-1-0_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/libpolkit-backend-1-0_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/linux-image-4.4.0-1074-aws_4.4.0-1074.84_amd64.deb /var/cache/apt/archives/linux-aws_4.4.0.1074.76_amd64.deb /var/cache/apt/archives/linux-image-aws_4.4.0.1074.76_amd64.deb /var/cache/apt/archives/linux-aws-headers-4.4.0-1074_4.4.0-1074.84_all.deb /var/cache/apt/archives/linux-headers-4.4.0-1074-aws_4.4.0-1074.84_amd64.deb /var/cache/apt/archives/linux-headers-aws_4.4.0.1074.76_amd64.deb /var/cache/apt/archives/policykit-1_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/python3-cryptography_1.2.3-1ubuntu0.2_amd64.deb /var/cache/apt/archives/python3-jwt_1.3.0-1ubuntu0.1_all.deb /var/cache/apt/archives/python3-requests_2.9.1-3ubuntu0.1_all.deb /var/cache/apt/archives/unattended-upgrades_0.90ubuntu0.10_all.deb root 12555 0.0 0.0 4500 740 pts/1 S+ 23:24 0:00 \_ /bin/sh /var/lib/dpkg/info/unattended-upgrades.prerm upgrade 0.90ubuntu0.10 root 12556 0.0 0.0 4500 1736 pts/1 S+ 23:24 0:00 \_ /bin/sh /usr/sbin/invoke-rc.d unattended-upgrades stop root 12594 0.0 0.0 24884 1388 pts/1 S+ 23:24 0:00 \_ systemctl stop unattended-upgrades.service
After about 15 minutes, the retry to install openjdk-8-jdk succeeded.
I'm not sure how to make this better except rebuild AMIs with updated packaging. Over time, the system's automatic apt-get update / upgrades are going to take longer and longer and our own stuff is going to block for longer and longer.
Attachments
Issue Links
- is related to
-
IMPALA-9319 Automate keeping build images up to date on jenkins.impala.io
- Open
- relates to
-
IMPALA-8767 Jenkins failures due to "Could not get lock /var/lib/dpkg/lock"
- Resolved