Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8115

some jenkins workers slow to do real work due to dpkg lock conflicts

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 3.4.0
    • Infrastructure
    • None
    • ghx-label-4

    Description

      A Jenkins worker for label ubuntu-16.04 took about 15 minutes to start doing real work. I noticed that it was retrying apt-get update:

      ++ sudo apt-get --yes install openjdk-8-jdk
      E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
      E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
      ++ date
      Thu Jan 24 23:37:33 UTC 2019
      ++ sudo apt-get update
      ++ sleep 10
      ++ sudo apt-get --yes install openjdk-8-jdk
      [etc]
      

      I ssh'd into a host and saw that, yes, something else was holding onto the dpkg log (confirmed with lsof and not pasted here. dpkg process PID 11459 was the culprit)

      root       1750  0.0  0.0   4508  1664 ?        Ss   23:21   0:00 /bin/sh /usr/lib/apt/apt.systemd.daily
      root       1804 12.3  0.1 141076 80452 ?        S    23:22   1:24  \_ /usr/bin/python3 /usr/bin/unattended-upgrade
      root       3263  0.0  0.1 140960 72896 ?        S    23:23   0:00      \_ /usr/bin/python3 /usr/bin/unattended-upgrade
      root      11459  0.6  0.0  45920 25184 pts/1    Ss+  23:24   0:03          \_ /usr/bin/dpkg --status-fd 10 --unpack --auto-deconfigure /var/cache/apt/archives/tzdata_2018i-0ubuntu0.16.04_all.deb /var/cache/apt/archives/distro-info-data_0.28ubuntu0.9_all.deb /var/cache/apt/archives/file_1%3a5.25-2ubuntu1.1_amd64.deb /var/cache/apt/archives/libmagic1_1%3a5.25-2ubuntu1.1_amd64.deb /var/cache/apt/archives/libisc-export160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libdns-export162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/isc-dhcp-client_4.3.3-5ubuntu12.9_amd64.deb /var/cache/apt/archives/isc-dhcp-common_4.3.3-5ubuntu12.9_amd64.deb /var/cache/apt/archives/libidn11_1.32-3ubuntu1.2_amd64.deb /var/cache/apt/archives/libpng12-0_1.2.54-1ubuntu1.1_amd64.deb /var/cache/apt/archives/libtasn1-6_4.7-3ubuntu0.16.04.3_amd64.deb /var/cache/apt/archives/libapparmor-perl_2.10.95-0ubuntu2.10_amd64.deb /var/cache/apt/archives/apparmor_2.10.95-0ubuntu2.10_amd64.deb /var/cache/apt/archives/curl_7.47.0-1ubuntu2.11_amd64.deb /var/cache/apt/archives/libgssapi-krb5-2_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libkrb5-3_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libkrb5support0_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libk5crypto3_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libcurl3-gnutls_7.47.0-1ubuntu2.11_amd64.deb /var/cache/apt/archives/apt-transport-https_1.2.29ubuntu0.1_amd64.deb /var/cache/apt/archives/libicu55_55.1-7ubuntu0.4_amd64.deb /var/cache/apt/archives/libxml2_2.9.3+dfsg1-1ubuntu0.6_amd64.deb /var/cache/apt/archives/bind9-host_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/dnsutils_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libisc160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libdns162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libisccc140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libisccfg140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/liblwres141_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libbind9-140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/openssl_1.0.2g-1ubuntu4.14_amd64.deb /var/cache/apt/archives/ca-certificates_20170717~16.04.1_all.deb /var/cache/apt/archives/libasprintf0v5_0.19.7-2ubuntu3.1_amd64.deb /var/cache/apt/archives/gettext-base_0.19.7-2ubuntu3.1_amd64.deb /var/cache/apt/archives/krb5-locales_1.13.2+dfsg-5ubuntu2.1_all.deb /var/cache/apt/archives/libelf1_0.165-3ubuntu1.1_amd64.deb /var/cache/apt/archives/libglib2.0-data_2.48.2-0ubuntu4.1_all.deb /var/cache/apt/archives/libnuma1_2.0.11-1ubuntu1.1_amd64.deb /var/cache/apt/archives/libpolkit-gobject-1-0_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/libx11-data_2%3a1.6.3-1ubuntu2.1_all.deb /var/cache/apt/archives/libx11-6_2%3a1.6.3-1ubuntu2.1_amd64.deb /var/cache/apt/archives/openssh-sftp-server_1%3a7.2p2-4ubuntu2.6_amd64.deb /var/cache/apt/archives/openssh-server_1%3a7.2p2-4ubuntu2.6_amd64.deb /var/cache/apt/archives/openssh-client_1%3a7.2p2-4ubuntu2.6_amd64.deb /var/cache/apt/archives/rsync_3.1.1-3ubuntu1.2_amd64.deb /var/cache/apt/archives/tcpdump_4.9.2-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/wget_1.17.1-1ubuntu1.4_amd64.deb /var/cache/apt/archives/python3-problem-report_2.20.1-0ubuntu2.18_all.deb /var/cache/apt/archives/python3-apport_2.20.1-0ubuntu2.18_all.deb /var/cache/apt/archives/apport_2.20.1-0ubuntu2.18_all.deb /var/cache/apt/archives/dns-root-data_2018013001~16.04.1_all.deb /var/cache/apt/archives/dnsmasq-base_2.75-1ubuntu0.16.04.5_amd64.deb /var/cache/apt/archives/git-man_1%3a2.7.4-0ubuntu1.6_all.deb /var/cache/apt/archives/git_1%3a2.7.4-0ubuntu1.6_amd64.deb /var/cache/apt/archives/libmspack0_0.5-1ubuntu0.16.04.3_amd64.deb /var/cache/apt/archives/libpolkit-agent-1-0_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/libpolkit-backend-1-0_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/linux-image-4.4.0-1074-aws_4.4.0-1074.84_amd64.deb /var/cache/apt/archives/linux-aws_4.4.0.1074.76_amd64.deb /var/cache/apt/archives/linux-image-aws_4.4.0.1074.76_amd64.deb /var/cache/apt/archives/linux-aws-headers-4.4.0-1074_4.4.0-1074.84_all.deb /var/cache/apt/archives/linux-headers-4.4.0-1074-aws_4.4.0-1074.84_amd64.deb /var/cache/apt/archives/linux-headers-aws_4.4.0.1074.76_amd64.deb /var/cache/apt/archives/policykit-1_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/python3-cryptography_1.2.3-1ubuntu0.2_amd64.deb /var/cache/apt/archives/python3-jwt_1.3.0-1ubuntu0.1_all.deb /var/cache/apt/archives/python3-requests_2.9.1-3ubuntu0.1_all.deb /var/cache/apt/archives/unattended-upgrades_0.90ubuntu0.10_all.deb
      root      12555  0.0  0.0   4500   740 pts/1    S+   23:24   0:00              \_ /bin/sh /var/lib/dpkg/info/unattended-upgrades.prerm upgrade 0.90ubuntu0.10
      root      12556  0.0  0.0   4500  1736 pts/1    S+   23:24   0:00                  \_ /bin/sh /usr/sbin/invoke-rc.d unattended-upgrades stop
      root      12594  0.0  0.0  24884  1388 pts/1    S+   23:24   0:00                      \_ systemctl stop unattended-upgrades.service
      

      After about 15 minutes, the retry to install openjdk-8-jdk succeeded.

      I'm not sure how to make this better except rebuild AMIs with updated packaging. Over time, the system's automatic apt-get update / upgrades are going to take longer and longer and our own stuff is going to block for longer and longer.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            laszlog Laszlo Gaal
            mikeb Michael Brown
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment