[MESOS-9899] Using a symlink as the agent's work directory results in non-removal of persistent volume mounts.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.6.0, 1.7.0, 1.8.0
• Labels: containerization containerizer gc
• Story Points: 2
• Assignee: Unassigned
• Reporter: Joseph Wu
• Created: 19/Jul/19 17:59
• Updated: 22/Aug/19 00:24

Description

The directory layout of the agent's information places created persistent volumes under the agent's --work_dir:

//   root ('--work_dir' flag)
//   |-- volumes
//   |   |-- roles
//   |       |-- <role>
//   |           |-- <persistence_id> (persistent volume)

When these persistent volumes are used, they will (on Linux) generally be mounted underneath the sandbox directory (also located under --work_dir). Upon termination of use, persistent volumes are unmounted by reading the mount table, and checking if any mount targets are under the sandbox:

  // Reverse unmount order to handle nested mount points.
  foreach (const fs::MountInfoTable::Entry& entry,
           adaptor::reverse(table->entries)) {
    // NOTE: All persistent volumes are mounted at targets under the
    // container's work directory. We unmount all the persistent
    // volumes before unmounting the sandbox/work directory mount.
    if (strings::startsWith(entry.target, sandbox)) {
      LOG(INFO) << "Unmounting volume '" << entry.target
                << "' for container " << containerId;

—

However, when an agent's work directory is placed under a symlink, the same code above might not find any persistent volumes to remove. This is because the mount table shows the real location on disk, but the sandbox expects the symlinked location.

For example, suppose:

• The --work_dir is /var/run/mesos.
• /var/run/mesos is a symlink pointing to /tmp/link.

The agent will create sandboxes under paths like /var/run/mesos/slave/.../framework/.../.... The mount table however, will show mount targets like /tmp/link/slave/.../framework/.../.... Since the mount table target does not start with the sandbox path, the filesystem/linux isolator will not find any persistent volumes to clean up. The agent's garbage collector will also fail here, because it tries to unmount any persistent volumes under the agent's work directory.

      foreach (const fs::MountInfoTable::Entry& entry,
               adaptor::reverse(mountTable->entries)) {
        // Ignore mounts whose targets are not under `workDir`.
        if (!strings::startsWith(
                path::join(entry.target, ""),
                path::join(_workDir, ""))) {
                continue;
        }