[MESOS-9352] Data in persistent volume deleted accidentally when using Docker container and Persistent volume

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: 1.5.1, 1.5.2
• Component/s: agent, containerization, docker
• Labels: dcos dcos-1.11.6 mesosphere persistent-volumes

Environment

DCOS 1.11.6

Mesos 1.5.2

• Sprint: Containerization RI12 Sp 43, Containerization: RI-13 Sp 44, Containerization: RI13 Sp 45, Containerization: RI14 Sp 46, Containerization: RI14 Sp 47
• Story Points: 5
• Assignee: Joseph Wu
• Reporter: David Ko
• Created: 24/Oct/18 14:26
• Updated: 22/Jul/21 06:47
• Resolved: 22/Jul/21 06:47

Description

Using docker image w/ persistent volume to start a service, it will cause data in persistent volume deleted accidentally when task killed and restarted, also old mount points not unmounted, even the service already deleted. 

The expected result should be data in persistent volume kept until task deleted completely, also dangling mount points should be unmounted correctly.

 

Step 1: Use below JSON config to create a Mysql server using Docker image and Persistent Volume

{
  "env": {
    "MYSQL_USER": "wordpress",
    "MYSQL_PASSWORD": "secret",
    "MYSQL_ROOT_PASSWORD": "supersecret",
    "MYSQL_DATABASE": "wordpress"
  },
  "id": "/mysqlgc",
  "backoffFactor": 1.15,
  "backoffSeconds": 1,
  "constraints": [
    [
      "hostname",
      "IS",
      "172.27.12.216"
    ]
  ],
  "container": {
    "portMappings": [
      {
        "containerPort": 3306,
        "hostPort": 0,
        "protocol": "tcp",
        "servicePort": 10000
      }
    ],
    "type": "DOCKER",
    "volumes": [
      {
        "persistent": {
          "type": "root",
          "size": 1000,
          "constraints": []
        },
        "mode": "RW",
        "containerPath": "mysqldata"
      },
      {
        "containerPath": "/var/lib/mysql",
        "hostPath": "mysqldata",
        "mode": "RW"
      }
    ],
    "docker": {
      "image": "mysql",
      "forcePullImage": false,
      "privileged": false,
      "parameters": []
    }
  },
  "cpus": 1,
  "disk": 0,
  "instances": 1,
  "maxLaunchDelaySeconds": 3600,
  "mem": 512,
  "gpus": 0,
  "networks": [
    {
      "mode": "container/bridge"
    }
  ],
  "residency": {
    "relaunchEscalationTimeoutSeconds": 3600,
    "taskLostBehavior": "WAIT_FOREVER"
  },
  "requirePorts": false,
  "upgradeStrategy": {
    "maximumOverCapacity": 0,
    "minimumHealthCapacity": 0
  },
  "killSelection": "YOUNGEST_FIRST",
  "unreachableStrategy": "disabled",
  "healthChecks": [],
  "fetch": []
}

Step 2: Kill mysqld process to force rescheduling new Mysql task, but found 2 mount points to the same persistent volume, it means old mount point did not be unmounted immediately.

Step 3: After GC, data in persistent volume was deleted accidentally, but mysqld (Mesos task) still running

Step 4: Delete Mysql service from Marathon, all mount points unable to unmount, even the service already deleted.

Issue Links

• is related to: MESOS-8830 Agent gc on old slave sandboxes could empty persistent volume data
• is related to: MESOS-9049 Agent GC could unmount a dangling persistent volume multiple times.
• is related to: MESOS-2408 Slave should reclaim storage for destroyed persistent volumes.