[MESOS-9049] Agent GC could unmount a dangling persistent volume multiple times.

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.4.2, 1.5.2, 1.6.1, 1.7.0
• Fix Version/s: 1.4.2, 1.5.2, 1.6.1, 1.7.0
• Component/s: agent
• Target Version/s: 1.4.2, 1.5.2, 1.6.1, 1.7.0
• Story Points: 2
• Assignee: Zhitao Li
• Reporter: Chun-Hung Hsiao
• Shepherd: Chun-Hung Hsiao
• Created: 03/Jul/18 20:08
• Updated: 24/Oct/18 19:12
• Resolved: 03/Jul/18 23:06

Description

When the agent GC an executor dir and the sandbox of one of its run that contains a dangling persistent volume, the agent might try to unmount the persistent volume twice, which leads to an EINVAL when trying to unmount the target for the second time.

Here is the log from a failure run of GarbageCollectorIntegrationTest.ROOT_DanglingMount:

W0702 23:35:31.669946 25401 gc.cpp:241] Unmounting dangling mount point '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000/executors/test-task123/runs/3fcde2c8-b461-4f22-afec-daa269291c95/dangling' of persistent volume '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/volumes/roles/default-role/persistence-id' inside garbage collected path '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000/executors/test-task123'
W0702 23:35:31.683878 25401 gc.cpp:241] Unmounting dangling mount point '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000/executors/test-task123/runs/3fcde2c8-b461-4f22-afec-daa269291c95/dangling' of persistent volume '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/volumes/roles/default-role/persistence-id' inside garbage collected path '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000'
W0702 23:35:31.683912 25401 gc.cpp:248] Skipping deletion of '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000' because unmount failed on '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000/executors/test-task123/runs/3fcde2c8-b461-4f22-afec-daa269291c95/dangling': Failed to unmount '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000/executors/test-task123/runs/3fcde2c8-b461-4f22-afec-daa269291c95/dangling': Invalid argument

Issue Links

• relates to: MESOS-9352 Data in persistent volume deleted accidentally when using Docker container and Persistent volume