Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9977

Agent does not check for immutable files while removing persistent volumes (and possibly in other GC operations)

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Accepted
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.6.2, 1.7.2, 1.8.1, 1.9.0
    • Fix Version/s: None
    • Component/s: agent
    • Labels:
    • Story Points:
      3

      Description

      We observed an exit/crash loop on an agent originating from deleting a persistent volume:

      slave.cpp:4557] Deleting persistent volume '<UUID>' at '/path/to/mesos/slave/volumes/roles/my-role/<UUID>'
      

      This persistent volume happened to have one (or more) files within marked as immutable.

      When the agent went to delete this persistent volume, via os::rmdir(...), it encountered these immutable file(s) and exits like:

      slave.cpp:4423] EXIT with status 1: Failed to sync checkpointed resources: Failed to remove persistent volume '<UUID>' at '/path/to/mesos/slave/volumes/roles/my-role/<UUID>': Operation not permitted
      

      The agent would then be unable to start up again, because during recovery, the agent would attempt to delete the same persistent volume and fail to do so.

      Manually removing the immutable attribute from files within the persistent volume allows the agent to recover:

      chattr -R -i /path/to/mesos/slave/volumes/roles/my-role/<UUID>
      

      Immutable attributes can be easily introduced by any tasks running on the agent. As long as the task has sufficient permissions, it could easily call chattr +i .... This attribute could also affect sandbox GC, which also uses os::rmdir to clean up. However, sandbox GC tends to warn rather than exit on failure.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                kaysoky Joseph Wu
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: