[MESOS-10131] Agent frequently dies with error "Cycle found in mount table hierarchy"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 1.9.0
• Component/s: agent, framework
• Assignee: Andrei Budnik
• Reporter: Thomas Plummer
• Created: 21/May/20 19:18
• Updated: 14/Apr/21 16:54
• Resolved: 14/Apr/21 16:54

Description

Our mesos agent frequently dies with the follow error in the slave logs:

 

F0509 22:10:33.036993 17723 fs.cpp:217] Check failed: !visitedParents.contains(parentId) Cycle found in mount table hierarchy at entry '1954': 
18 41 0:18 / /sys rw,nosuid,nodev,noexec,relatime shared:6 - sysfs sysfs rw,seclabel
19 41 0:3 / /proc rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
20 41 0:5 / /dev rw,nosuid shared:2 - devtmpfs devtmpfs rw,seclabel,size=65852208k,nr_inodes=16463052,mode=755
21 18 0:17 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:7 - securityfs securityfs rw
22 20 0:19 / /dev/shm rw,nosuid,nodev,noexec shared:3 - tmpfs tmpfs rw,seclabel
23 20 0:12 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=000
24 41 0:20 / /run rw,nosuid,nodev shared:24 - tmpfs tmpfs rw,seclabel,mode=755
25 18 0:21 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:8 - tmpfs tmpfs ro,seclabel,mode=755
26 25 0:22 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:9 - cgroup cgroup rw,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
27 18 0:23 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw
28 18 0:24 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:21 - efivarfs efivarfs rw
29 25 0:25 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:10 - cgroup cgroup rw,seclabel,perf_event
30 25 0:26 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:11 - cgroup cgroup rw,seclabel,net_prio,net_cls
31 25 0:27 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:12 - cgroup cgroup rw,seclabel,cpuset
32 25 0:28 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:13 - cgroup cgroup rw,seclabel,blkio
33 25 0:29 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:14 - cgroup cgroup rw,seclabel,freezer
34 25 0:30 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,seclabel,hugetlb
35 25 0:31 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,seclabel,devices
36 25 0:32 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,seclabel,cpuacct,cpu
37 25 0:33 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,seclabel,memory
38 25 0:34 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,seclabel,pids
39 18 0:35 / /sys/kernel/config rw,relatime shared:22 - configfs configfs rw
41 0 253:0 / / rw,relatime shared:1 - xfs /dev/mapper/vg_system-root rw,seclabel,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota
42 18 0:16 / /sys/fs/selinux rw,relatime shared:23 - selinuxfs selinuxfs rw
43 19 0:37 / /proc/sys/fs/binfmt_misc rw,relatime shared:25 - autofs systemd-1 rw,fd=32,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11414
44 18 0:6 / /sys/kernel/debug rw,relatime shared:26 - debugfs debugfs rw
45 20 0:15 / /dev/mqueue rw,relatime shared:27 - mqueue mqueue rw,seclabel
46 20 0:38 / /dev/hugepages rw,relatime shared:28 - hugetlbfs hugetlbfs rw,seclabel
47 41 8:2 / /boot rw,relatime shared:29 - xfs /dev/sda2 rw,seclabel,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota
48 47 8:1 / /boot/efi rw,relatime shared:30 - vfat /dev/sda1 rw,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro
49 41 253:2 / /var rw,relatime shared:31 - xfs /dev/mapper/vg_system-var rw,seclabel,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota
50 41 253:5 / /home rw,nodev,relatime shared:32 - xfs /dev/mapper/vg_system-home rw,seclabel,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota
51 41 253:4 / /tmp rw,nosuid,nodev,noexec,relatime shared:33 - xfs /dev/mapper/vg_system-tmp rw,seclabel,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota
53 49 253:4 / /var/tmp rw,nosuid,nodev,noexec,relatime shared:33 - xfs /dev/mapper/vg_system-tmp rw,seclabel,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota
52 49 253:3 / /var/log rw,relatime shared:34 - xfs /dev/mapper/vg_system-varlog rw,seclabel,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota
54 52 253:6 / /var/log/audit rw,relatime shared:35 - xfs /dev/mapper/vg_system-varlogaudit rw,seclabel,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota
187 41 0:41 / /mnt/receipt rw,relatime shared:165 - nfs4 dtmetlnfsa01p.a.carfax.us:/ rw,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.18.154.117,local_lock=none,addr=172.18.138.237
188 41 0:42 / /mnt/receipt_web_dev rw,relatime shared:169 - nfs4 dtmetlnfsa01b.a.carfax.us:/ rw,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.18.154.117,local_lock=none,addr=172.18.137.248
192 41 0:41 / /mnt/receipt_web_prod rw,relatime shared:173 - nfs4 dtmetlnfsa01p.a.carfax.us:/ rw,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.18.154.117,local_lock=none,addr=172.18.138.237
204 49 0:44 / /var/lib/docker/overlay2/cb547c8d78b39d371bf7089f97d6e543ae288edf0eaedcc501d0b5bf43aa7678/merged rw,relatime shared:182 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/CQCC2HFSPJ7QHEOCMFVRWVI7ZO:/var/lib/docker/overlay2/l/JURMJLD7MENW4LDHYQELJI3GVU:/var/lib/docker/overlay2/l/65H3GZANPU7PXQKMZ5LYSX3NMD:/var/lib/docker/overlay2/l/EBLLCFPM7KMXOZP7HWZMUG2KDW,upperdir=/var/lib/docker/overlay2/cb547c8d78b39d371bf7089f97d6e543ae288edf0eaedcc501d0b5bf43aa7678/diff,workdir=/var/lib/docker/overlay2/cb547c8d78b39d371bf7089f97d6e543ae288edf0eaedcc501d0b5bf43aa7678/work
213 49 0:45 / /var/lib/docker/containers/2e6387919e2e2356ee135986346cd3ff21d8dba2b72cf2e34af7dc3b7df39bf5/mounts/shm rw,nosuid,nodev,noexec,relatime shared:186 - tmpfs shm rw,seclabel,size=65536k
289 24 0:3 / /run/docker/netns/7ad915a9c47b rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
219 49 0:55 / /var/lib/docker/overlay2/640baf9466471c301c4b756efe3a49a98364b3048cc5e67abc7ea402779ce483/merged rw,relatime shared:193 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/EWSSWVEZQMW3G7ZG326TSYCIGU:/var/lib/docker/overlay2/l/PZTVYDVFJNG6UWTYMCCBBVM7SJ:/var/lib/docker/overlay2/l/5Q7WXYOBICJJ7HMGXPUFQFTVUL:/var/lib/docker/overlay2/l/Y5KMKEPFKG4PISNVPL3WPFJZRR:/var/lib/docker/overlay2/l/6MCL5ODW4IDXI4GMQ3ASIJ335K:/var/lib/docker/overlay2/l/WMT4ZUFOXI22O5QND73MB7EXOZ:/var/lib/docker/overlay2/l/ZAYVC4PH3CJ2HN5JLCNEN3B2NS,upperdir=/var/lib/docker/overlay2/640baf9466471c301c4b756efe3a49a98364b3048cc5e67abc7ea402779ce483/diff,workdir=/var/lib/docker/overlay2/640baf9466471c301c4b756efe3a49a98364b3048cc5e67abc7ea402779ce483/work
243 49 0:56 / /var/lib/docker/containers/b92d035a27172061fb3fcdb46740bac39547a5889ffbfdb51c05a783b24e2c1e/mounts/shm rw,nosuid,nodev,noexec,relatime shared:197 - tmpfs shm rw,seclabel,size=65536k
352 24 0:3 / /run/docker/netns/ac1bfa3f059a rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
249 49 0:66 / /var/lib/docker/overlay2/1726b9d9ffaceedd9f9090265ac9d99dadc844a6e0c057985ab9da63a3a0446d/merged rw,relatime shared:204 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/I34T5H5YYJLUJUAHOAWED5YRDF:/var/lib/docker/overlay2/l/SPDY4UXXR7XFNDEMLJWTWDSHWH:/var/lib/docker/overlay2/l/6JVIPP3XCCWKZPFAUWKXCDWYXL:/var/lib/docker/overlay2/l/L5VKHJHVOWG24VJPJCAKGTQX5G:/var/lib/docker/overlay2/l/ZIIS5MWCIF4C6KXI2LVKVU4TMF:/var/lib/docker/overlay2/l/4JXIZ5TFXIHYPXDCCGHBZTZ72Q:/var/lib/docker/overlay2/l/6XUWKFR4KNTTXGCV4G7EZ4GDJ4:/var/lib/docker/overlay2/l/WJJ556LZD7CIZYR4J6L2QM66CA:/var/lib/docker/overlay2/l/XBZVPQOAJYXYNDI4TDO3P6LAM2,upperdir=/var/lib/docker/overlay2/1726b9d9ffaceedd9f9090265ac9d99dadc844a6e0c057985ab9da63a3a0446d/diff,workdir=/var/lib/docker/overlay2/1726b9d9ffaceedd9f9090265ac9d99dadc844a6e0c057985ab9da63a3a0446d/work
306 49 0:67 / /var/lib/docker/containers/5d3d5a5b4fea101faaf1d6b7a8c238ce65259d498fc1da641245ccdb27f473b4/mounts/shm rw,nosuid,nodev,noexec,relatime shared:208 - tmpfs shm rw,seclabel,size=65536k
418 24 0:3 / /run/docker/netns/97a9ba9365e5 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
353 49 0:77 / /var/lib/docker/overlay2/e9a44d7cc10346f46494e30af013ee62f3bd0578ba85dae6343d3e19efa765fa/merged rw,relatime shared:215 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/NYZCGL4KRZPMOC47ZDW6WGHPSA:/var/lib/docker/overlay2/l/FXX54ZOYNXLTW44Y5L6CRJHTDG:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/e9a44d7cc10346f46494e30af013ee62f3bd0578ba85dae6343d3e19efa765fa/diff,workdir=/var/lib/docker/overlay2/e9a44d7cc10346f46494e30af013ee62f3bd0578ba85dae6343d3e19efa765fa/work
369 49 0:78 / /var/lib/docker/containers/313abaf8c9008d97b92bdbed4ebe176a16c3a1a470d043356f687f23b5f6e064/mounts/shm rw,nosuid,nodev,noexec,relatime shared:219 - tmpfs shm rw,seclabel,size=65536k
489 24 0:3 / /run/docker/netns/457d0825b88e rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
200 49 0:43 / /var/lib/docker/overlay2/9b1171e70f1b91d10b1dd88eff9b955f1622b4e37b677141d3750d3be326de37/merged rw,relatime shared:177 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/CJCJSATXHYGFQR7U6DGA5KCFNH:/var/lib/docker/overlay2/l/CGQQKI7B6AYE2V5UUBR3LBMLHU:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/9b1171e70f1b91d10b1dd88eff9b955f1622b4e37b677141d3750d3be326de37/diff,workdir=/var/lib/docker/overlay2/9b1171e70f1b91d10b1dd88eff9b955f1622b4e37b677141d3750d3be326de37/work
426 24 0:3 / /run/docker/netns/default rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
430 49 0:88 / /var/lib/docker/containers/0cbdc0832d7171b1fe438b524218aa69f25be89e342b733a35da6392dde6f3b6/mounts/shm rw,nosuid,nodev,noexec,relatime shared:228 - tmpfs shm rw,seclabel,size=65536k
490 49 0:98 / /var/lib/docker/overlay2/e45051b2b9045d47ef8d9986f228f326f6eb278911a7bc2df28abd8d839957ae/merged rw,relatime shared:232 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/JF6XYZASTGJPMMI4ATLVL7XQLN:/var/lib/docker/overlay2/l/XH7BYPZETAJZACKGSIE2UPFTYT:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/e45051b2b9045d47ef8d9986f228f326f6eb278911a7bc2df28abd8d839957ae/diff,workdir=/var/lib/docker/overlay2/e45051b2b9045d47ef8d9986f228f326f6eb278911a7bc2df28abd8d839957ae/work
436 49 0:97 / /var/lib/docker/overlay2/4f3d53bdf602862a913ae5aa0c3bb3f8b3201880a385fd537866a4875c943118/merged rw,relatime shared:236 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/Z4736YBPMM7IVNYRMTQLABNMLO:/var/lib/docker/overlay2/l/NTZPW5OZN7G2XWHA2A2HODZN4H:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/4f3d53bdf602862a913ae5aa0c3bb3f8b3201880a385fd537866a4875c943118/diff,workdir=/var/lib/docker/overlay2/4f3d53bdf602862a913ae5aa0c3bb3f8b3201880a385fd537866a4875c943118/work
509 49 0:99 / /var/lib/docker/containers/568a322d9de92d692f44c89edbb247f99233c2b4eaf2c6e174834f5e55631aa5/mounts/shm rw,nosuid,nodev,noexec,relatime shared:240 - tmpfs shm rw,seclabel,size=65536k
604 49 0:102 / /var/lib/docker/containers/01837e377d2f3f2f8789009e7a2756c635090f96ccf273fe1c0a57266adb8503/mounts/shm rw,nosuid,nodev,noexec,relatime shared:244 - tmpfs shm rw,seclabel,size=65536k
724 24 0:3 / /run/docker/netns/1d9331c46297 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
325 24 0:3 / /run/docker/netns/fac4bd7ceba4 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
580 49 0:120 / /var/lib/docker/overlay2/5738c6c223a279e49f68ebc5d33fd1c89f3ce681c36dc7a1edc539d6bc8b17d1/merged rw,relatime shared:254 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/3GUXKFN35HRVUVQPOVPM4TVDNI:/var/lib/docker/overlay2/l/QEGAB2XRUOFX6SDP2T6HAG7YTQ:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/5738c6c223a279e49f68ebc5d33fd1c89f3ce681c36dc7a1edc539d6bc8b17d1/diff,workdir=/var/lib/docker/overlay2/5738c6c223a279e49f68ebc5d33fd1c89f3ce681c36dc7a1edc539d6bc8b17d1/work
515 49 0:119 / /var/lib/docker/overlay2/927421db2a97c6b97fa91c02924feb71664ca817687df193af6cd1a5ae6b50a0/merged rw,relatime shared:258 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/W76CZVYMUATPXVS6SQZSZCF2WZ:/var/lib/docker/overlay2/l/V7QP6E33UFHAFCTBCW6YX7UKAN:/var/lib/docker/overlay2/l/6JVIPP3XCCWKZPFAUWKXCDWYXL:/var/lib/docker/overlay2/l/L5VKHJHVOWG24VJPJCAKGTQX5G:/var/lib/docker/overlay2/l/ZIIS5MWCIF4C6KXI2LVKVU4TMF:/var/lib/docker/overlay2/l/4JXIZ5TFXIHYPXDCCGHBZTZ72Q:/var/lib/docker/overlay2/l/6XUWKFR4KNTTXGCV4G7EZ4GDJ4:/var/lib/docker/overlay2/l/WJJ556LZD7CIZYR4J6L2QM66CA:/var/lib/docker/overlay2/l/XBZVPQOAJYXYNDI4TDO3P6LAM2,upperdir=/var/lib/docker/overlay2/927421db2a97c6b97fa91c02924feb71664ca817687df193af6cd1a5ae6b50a0/diff,workdir=/var/lib/docker/overlay2/927421db2a97c6b97fa91c02924feb71664ca817687df193af6cd1a5ae6b50a0/work
593 49 0:121 / /var/lib/docker/containers/e22b016ba4f9713ad1744861f2080564f39340c05a69aeb232fee788599a1856/mounts/shm rw,nosuid,nodev,noexec,relatime shared:262 - tmpfs shm rw,seclabel,size=65536k
597 49 0:122 / /var/lib/docker/containers/ccb2fb05de53c0420bdfd01a45ed7fc75b14bd40381b394941b60fd7e365a133/mounts/shm rw,nosuid,nodev,noexec,relatime shared:266 - tmpfs shm rw,seclabel,size=65536k
856 24 0:3 / /run/docker/netns/db9bcbe91e60 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
862 24 0:3 / /run/docker/netns/a48ae2ca4c83 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
636 49 0:141 / /var/lib/docker/overlay2/d9a1f696a9a1a0a367cb69ade4f97f0cc020b6d1ecace518f683715909e775b6/merged rw,relatime shared:276 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/QWSEO4MRSR3C32FKLS7PLIVRPJ:/var/lib/docker/overlay2/l/243LDQSMXQ473GU7OYIWZP77RM:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/d9a1f696a9a1a0a367cb69ade4f97f0cc020b6d1ecace518f683715909e775b6/diff,workdir=/var/lib/docker/overlay2/d9a1f696a9a1a0a367cb69ade4f97f0cc020b6d1ecace518f683715909e775b6/work
673 49 0:142 / /var/lib/docker/containers/582b99472949822ac225cb49c969ef321b87f6ae17095614cac0c8bbd606807e/mounts/shm rw,nosuid,nodev,noexec,relatime shared:280 - tmpfs shm rw,seclabel,size=65536k
679 49 0:151 / /var/lib/docker/overlay2/f7367e73064bb0344c634823f0fcf8bb824acbadd97b50ca63e131d70ce81865/merged rw,relatime shared:284 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/ADJ2D3YKHYI7CIYPHDERVZ33T2:/var/lib/docker/overlay2/l/LXT64FCZASUYXSGLKQD5MQNL7I:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/f7367e73064bb0344c634823f0fcf8bb824acbadd97b50ca63e131d70ce81865/diff,workdir=/var/lib/docker/overlay2/f7367e73064bb0344c634823f0fcf8bb824acbadd97b50ca63e131d70ce81865/work
767 49 0:152 / /var/lib/docker/containers/024f22a36159cf2f890d9657a8a9131fe14b3b5a985a639afa5076e1fba2038b/mounts/shm rw,nosuid,nodev,noexec,relatime shared:288 - tmpfs shm rw,seclabel,size=65536k
937 24 0:3 / /run/docker/netns/de129995958c rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
874 49 0:164 / /var/lib/docker/overlay2/866931df90693dcf5ea2d3c4747c9496646c631ade2c82e392cd102d0bcdd629/merged rw,relatime shared:295 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/23F3FKUINKPFOWPY7DZL5ERHU4:/var/lib/docker/overlay2/l/EEXUD7ENX6EOSFLQSSBXA5MJBT:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/866931df90693dcf5ea2d3c4747c9496646c631ade2c82e392cd102d0bcdd629/diff,workdir=/var/lib/docker/overlay2/866931df90693dcf5ea2d3c4747c9496646c631ade2c82e392cd102d0bcdd629/work
797 49 0:163 / /var/lib/docker/overlay2/bc9a32c17146b0da8c492384ba4d6b278b8ec3c9b576577fa9cf9bbc3df79449/merged rw,relatime shared:299 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/VO2I6YDZ5DAJNDL52MQ3EZ5CY3:/var/lib/docker/overlay2/l/EEXUD7ENX6EOSFLQSSBXA5MJBT:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/bc9a32c17146b0da8c492384ba4d6b278b8ec3c9b576577fa9cf9bbc3df79449/diff,workdir=/var/lib/docker/overlay2/bc9a32c17146b0da8c492384ba4d6b278b8ec3c9b576577fa9cf9bbc3df79449/work
945 49 0:167 / /var/lib/docker/overlay2/18e2ba68cd9d04ef5a9d5df53c174ff1acad926422a25ccd8ca57026a0bd45bf/merged rw,relatime shared:307 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/PPCEVOPV2OOICEB3Q3QLAEUBG6:/var/lib/docker/overlay2/l/EEXUD7ENX6EOSFLQSSBXA5MJBT:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/18e2ba68cd9d04ef5a9d5df53c174ff1acad926422a25ccd8ca57026a0bd45bf/diff,workdir=/var/lib/docker/overlay2/18e2ba68cd9d04ef5a9d5df53c174ff1acad926422a25ccd8ca57026a0bd45bf/work
973 49 0:169 / /var/lib/docker/containers/a5d5176525f2ea32c26e7bdac0bfaa972b78d1865e3c45c89e7855c3a58965d9/mounts/shm rw,nosuid,nodev,noexec,relatime shared:319 - tmpfs shm rw,seclabel,size=65536k
977 49 0:170 / /var/lib/docker/containers/b3aa295350bf708b6a00d6cb0f4692debe8ff4db770874758062424b399b38ec/mounts/shm rw,nosuid,nodev,noexec,relatime shared:327 - tmpfs shm rw,seclabel,size=65536k
1209 24 0:3 / /run/docker/netns/201bae350626 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
1215 49 0:184 / /var/lib/docker/containers/77b71858d9bd7d5654fb4c7f8d6fe57e374f3ea60ab034ce12b2be3dbcb761ef/mounts/shm rw,nosuid,nodev,noexec,relatime shared:338 - tmpfs shm rw,seclabel,size=65536k
1309 24 0:3 / /run/docker/netns/1949215d3df9 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
1132 24 0:3 / /run/docker/netns/13f2b8b0cbb8 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
1163 49 0:250 / /var/lib/docker/overlay2/b6de234de6722bff9b65fa28e6197d9aba6718d6b90bee8bd6a4aafc6c262cb2/merged rw,relatime shared:383 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/UKMEBUO64YTYVEALHZ6J36IAMT:/var/lib/docker/overlay2/l/TTVIKPBU6ZLBC22QHFJ2B6GIC2:/var/lib/docker/overlay2/l/PZP7AK3R2XF5NX5JAJZMYTZL57:/var/lib/docker/overlay2/l/PESCRJUSG3NHQL7C7EIWECVCCF:/var/lib/docker/overlay2/l/SCBVAO3KAKUNJOPQN26GVSYCG6:/var/lib/docker/overlay2/l/63U5OLGIXU45AN2Q66R7AH2K44:/var/lib/docker/overlay2/l/CIKKQQYDWDU753HH6AXNUQEU4M:/var/lib/docker/overlay2/l/UBNGOAYVKTFIOFWPDCF2YVWKV6:/var/lib/docker/overlay2/l/WHT6BIFNOUT4ZKT7HD4M57BHOC,upperdir=/var/lib/docker/overlay2/b6de234de6722bff9b65fa28e6197d9aba6718d6b90bee8bd6a4aafc6c262cb2/diff,workdir=/var/lib/docker/overlay2/b6de234de6722bff9b65fa28e6197d9aba6718d6b90bee8bd6a4aafc6c262cb2/work
1244 49 0:251 / /var/lib/docker/containers/68504ddca0c41c080b878be86dad16d5bbcd5457b1ac8c9939e02e5c6b406903/mounts/shm rw,nosuid,nodev,noexec,relatime shared:387 - tmpfs shm rw,seclabel,size=65536k
1593 24 0:3 / /run/docker/netns/8b81b7d0d47c rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
1310 49 0:272 / /var/lib/docker/overlay2/d9dd935c27d71c16507ccd899049fe076dc98cbaa394c1aa5001bffacb24354e/merged rw,relatime shared:405 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/G5CAOULTRC2HC2G4E2JCKN3QW6:/var/lib/
docker/overlay2/l/LOG7DILAFLJBIQ7CKDQVFXJLP7:/var/lib/docker/overlay2/l/6JVIPP3XCCWKZPFAUWKXCDWYXL:/var/lib/docker/overlay2/l/L5VKHJHVOWG24VJPJCAKGTQX5G:/var/lib/docker/overlay2/l/ZIIS5MWCIF4C6KXI2LVKVU4TMF:/var/lib/docker/overlay2/l/4JXI
Z5TFXIHYPXDCCGHBZTZ72Q:/var/lib/docker/overlay2/l/6XUWKFR4KNTTXGCV4G7EZ4GDJ4:/var/lib/docker/overlay2/l/WJJ556LZD7CIZYR4J6L2QM66CA:/var/lib/docker/overlay2/l/XBZVPQOAJYXYNDI4TDO3P6LAM2,upperdir=/var/lib/docker/overlay2/d9dd935c27d71c16507
ccd899049fe076dc98cbaa394c1aa5001bffacb24354e/diff,workdir=/var/lib/docker/overlay2/d9dd935c27d71c16507ccd899049fe076dc98cbaa394c1aa5001bffacb24354e/work
1407 49 0:273 / /var/lib/docker/containers/60541340acc6f01ce4df0a6a84b7bc11c24e5a3841c6198703a0fcf56287f0b4/mounts/shm rw,nosuid,nodev,noexec,relatime shared:409 - tmpfs shm rw,seclabel,size=65536k
1686 24 0:3 / /run/docker/netns/4e6593953c78 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
1511 49 0:305 / /var/lib/docker/overlay2/3f9605f8bde1d6b263bc508ab72fd7921ef9bd845ec478a2789ee1649888a086/merged rw,relatime shared:438 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/WG33ZLCSKHFOWVRETKF2HSG24M:/var/lib/
docker/overlay2/l/OVCGDAOUN3T7CEJDVFVLEHGT3O:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZ
I6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/3f9605f8bde1d6b263bc508ab72fd7921ef9bd845ec478a2789ee1649888a086/diff,wor
kdir=/var/lib/docker/overlay2/3f9605f8bde1d6b263bc508ab72fd7921ef9bd845ec478a2789ee1649888a086/work
1642 49 0:306 / /var/lib/docker/containers/1d0fd3818c9c76e535ed7ef55da64734c37f80ed34a63e120fb9a37d3a799a19/mounts/shm rw,nosuid,nodev,noexec,relatime shared:442 - tmpfs shm rw,seclabel,size=65536k
1884 24 0:3 / /run/docker/netns/6d5fb91fd868 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
1249 49 0:294 / /var/lib/docker/overlay2/6124f7b96831698a1ffecb7a30f11c5ffd816acdf9cb2328cdf67295f11d144c/merged rw,relatime shared:427 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/V2TN4WZXNG2BU6NXWJVL7ZRQXR:/var/lib/
docker/overlay2/l/3YTRLN4RBALP2TRWPE3VYS424R:/var/lib/docker/overlay2/l/6JVIPP3XCCWKZPFAUWKXCDWYXL:/var/lib/docker/overlay2/l/L5VKHJHVOWG24VJPJCAKGTQX5G:/var/lib/docker/overlay2/l/ZIIS5MWCIF4C6KXI2LVKVU4TMF:/var/lib/docker/overlay2/l/4JXI
Z5TFXIHYPXDCCGHBZTZ72Q:/var/lib/docker/overlay2/l/6XUWKFR4KNTTXGCV4G7EZ4GDJ4:/var/lib/docker/overlay2/l/WJJ556LZD7CIZYR4J6L2QM66CA:/var/lib/docker/overlay2/l/XBZVPQOAJYXYNDI4TDO3P6LAM2,upperdir=/var/lib/docker/overlay2/6124f7b96831698a1ff
ecb7a30f11c5ffd816acdf9cb2328cdf67295f11d144c/diff,workdir=/var/lib/docker/overlay2/6124f7b96831698a1ffecb7a30f11c5ffd816acdf9cb2328cdf67295f11d144c/work
1506 49 0:295 / /var/lib/docker/containers/1e6c69bff96deee33cd6fd38ec3476fcc4dab3bd80a8d26412d0025adb5fee0f/mounts/shm rw,nosuid,nodev,noexec,relatime shared:431 - tmpfs shm rw,seclabel,size=65536k
2048 24 0:3 / /run/docker/netns/3847160c5290 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
1478 49 0:349 / /var/lib/docker/overlay2/7f94e23580df6e9d80d5d6dab2e60d618ab86d5ecaa86faffcd3dc3375db19e6/merged rw,relatime shared:482 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/XJ3SFXKEZQ5I7EKJYM4APE3UPQ:/var/lib/docker/overlay2/l/ABTA7UWBOUIOPJTFJWNCQ7UY5Z:/var/lib/docker/overlay2/l/F2UPYZKBQWD5GTBBG3SDVUQXFU:/var/lib/docker/overlay2/l/V7OVIKDLTSCRUWZILN5UICSAXW:/var/lib/docker/overlay2/l/5GUVZ4W5Z74C5FKXHG2HXV6HDV:/var/lib/docker/overlay2/l/3X56A4ZWENACGRZY3KTN7UKPAW,upperdir=/var/lib/docker/overlay2/7f94e23580df6e9d80d5d6dab2e60d618ab86d5ecaa86faffcd3dc3375db19e6/diff,workdir=/var/lib/docker/overlay2/7f94e23580df6e9d80d5d6dab2e60d618ab86d5ecaa86faffcd3dc3375db19e6/work
1804 49 0:350 / /var/lib/docker/containers/5311fa0cf04ac03f365134c72669ba340c874c90912f32c263b06559ba9e89f9/mounts/shm rw,nosuid,nodev,noexec,relatime shared:486 - tmpfs shm rw,seclabel,size=65536k
2141 24 0:3 / /run/docker/netns/6f5cee2e30ef rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
1810 49 0:360 / /var/lib/docker/overlay2/c5faf5ea2f781a13c3a80a8f6e47953832f1953a859fab42a7db1392f9dece04/merged rw,relatime shared:493 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/Z5U44MMVD6VKTT53JUEYKGERL6:/var/lib/docker/overlay2/l/JY6SAD525RZEX5UJ52S5BEK7MG:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/c5faf5ea2f781a13c3a80a8f6e47953832f1953a859fab42a7db1392f9dece04/diff,workdir=/var/lib/docker/overlay2/c5faf5ea2f781a13c3a80a8f6e47953832f1953a859fab42a7db1392f9dece04/work
1889 49 0:361 / /var/lib/docker/overlay2/42544267f5b3f8bdde983a2689c513d919f52eeb9664ed00081d4c13b7437756/merged rw,relatime shared:497 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/GPD2HTXNCCLSH2KXOULWKGMBMY:/var/lib/docker/overlay2/l/JY6SAD525RZEX5UJ52S5BEK7MG:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/42544267f5b3f8bdde983a2689c513d919f52eeb9664ed00081d4c13b7437756/diff,workdir=/var/lib/docker/overlay2/42544267f5b3f8bdde983a2689c513d919f52eeb9664ed00081d4c13b7437756/work
1902 49 0:362 / /var/lib/docker/containers/b7b21fb2213fb28eede63b1bb91c204368a119bebab82dded14cc2a8efe6c004/mounts/shm rw,nosuid,nodev,noexec,relatime shared:501 - tmpfs shm rw,seclabel,size=65536k
1906 49 0:363 / /var/lib/docker/containers/899b7704ee16f9059cbad240616597a4b2ba079c74be771930800019651eb47d/mounts/shm rw,nosuid,nodev,noexec,relatime shared:505 - tmpfs shm rw,seclabel,size=65536k
2386 24 0:3 / /run/docker/netns/df49e3594377 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
2392 24 0:3 / /run/docker/netns/d84efa1bac82 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
1912 49 0:385 / /var/lib/docker/overlay2/14c69840eaaec4ec9e0d6113e97c6ee194e8a7a8c45fdf744e53ad348e25455d/merged rw,relatime shared:519 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/2UB4ZYCNDCJM4BSGQNBGAHZD2T:/var/lib/docker/overlay2/l/POXAUWWDGN2EVRDC7OLXOQMVXW:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/14c69840eaaec4ec9e0d6113e97c6ee194e8a7a8c45fdf744e53ad348e25455d/diff,workdir=/var/lib/docker/overlay2/14c69840eaaec4ec9e0d6113e97c6ee194e8a7a8c45fdf744e53ad348e25455d/work
2197 49 0:389 / /var/lib/docker/containers/d52a0bf4e0b855aaba975b4c16af3eef2cf8c39fc0bd105d9e30d26b46b97585/mounts/shm rw,nosuid,nodev,noexec,relatime shared:538 - tmpfs shm rw,seclabel,size=65536k
3211 24 0:3 / /run/docker/netns/4ba229a552c5 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
177 49 0:39 / /var/lib/docker/overlay2/b229e0cc82584f4a73944909002413d6c6ba656a82eb2024e2db5464a66e2194/merged rw,relatime shared:155 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/NX6PSMOSP2RP6LWTWUIF6WJHVO:/var/lib/docker/overlay2/l/7CKFSNR2NSSSIFQLORD6PTXQFM:/var/lib/docker/overlay2/l/373RMBPEBVKCZPQAORFFAMBDBA:/var/lib/docker/overlay2/l/Z7NFRB4XSF2H3M2TW3UOMPTPOR:/var/lib/docker/overlay2/l/XODDIP677F2AVUQAYFE3OGKA4I:/var/lib/docker/overlay2/l/ZAEZI6G3PJKRYODIFJ6ZSPLVUH:/var/lib/docker/overlay2/l/U3KY4REW33LVKDOS2JH6C3RLYF:/var/lib/docker/overlay2/l/GJKGILWBQ2BFLH4IGQGM64KBHN,upperdir=/var/lib/docker/overlay2/b229e0cc82584f4a73944909002413d6c6ba656a82eb2024e2db5464a66e2194/diff,workdir=/var/lib/docker/overlay2/b229e0cc

This appears to be related to issue https://issues.apache.org/jira/browse/MESOS-6118

As reported in this issue we have a framework that runs a bunch of short lived jobs.

 

Issue Links

• Blocked: MESOS-6118 Agent would crash with docker container tasks due to host mount table read.