[MESOS-9196] Removing rootfs mounts may fail with EBUSY.

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.4.2, 1.5.1, 1.6.1
• Fix Version/s: 1.4.3, 1.5.2, 1.6.2, 1.7.0
• Component/s: containerization
• Labels: containerizer
• Target Version/s: 1.5.2, 1.6.2, 1.7.0
• Epic Link: Unified Container
• Sprint: Mesosphere Sprint 2018-28
• Story Points: 5
• Assignee: Jie Yu
• Reporter: Gilbert Song
• Shepherd: Gilbert Song
• Created: 30/Aug/18 22:36
• Updated: 10/Oct/18 23:58
• Resolved: 01/Sep/18 08:20

Description

We observed in production environment that this

Failed to destroy the provisioned rootfs when destroying container: Collect failed: Failed to destroy overlay-mounted rootfs '/var/lib/mesos/slave/provisioner/containers/6332cf3d-9897-475b-88b3-40e983a2a531/containers/e8f36ad7-c9ae-40da-9d14-431e98174735/backends/overlay/rootfses/d601ef1b-11b9-445a-b607-7c6366cd21ec': Failed to unmount '/var/lib/mesos/slave/provisioner/containers/6332cf3d-9897-475b-88b3-40e983a2a531/containers/e8f36ad7-c9ae-40da-9d14-431e98174735/backends/overlay/rootfses/d601ef1b-11b9-445a-b607-7c6366cd21ec': Device or resource busy

Consider fixing the issue by using detach unmount when unmounting container rootfs. See MESOS-3349 for details.

The root cause on why "Device or resource busy" is received when doing rootfs unmount is still unknown.

UPDATE: The production environment has a cronjob that scan filesystems to build index (updatedb for mlocate). This can explain the EBUSY we receive when doing `unmount`.

UPDATE: Splunk that's scanning `/var/lib/mesos` could also be a source of triggers.