Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9049

Agent GC could unmount a dangling persistent volume multiple times.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.2, 1.5.2, 1.6.1, 1.7.0
    • Fix Version/s: 1.4.2, 1.5.2, 1.6.1, 1.7.0
    • Component/s: agent
    • Labels:
      None

      Description

      When the agent GC an executor dir and the sandbox of one of its run that contains a dangling persistent volume, the agent might try to unmount the persistent volume twice, which leads to an EINVAL when trying to unmount the target for the second time.

      Here is the log from a failure run of GarbageCollectorIntegrationTest.ROOT_DanglingMount:

      W0702 23:35:31.669946 25401 gc.cpp:241] Unmounting dangling mount point '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000/executors/test-task123/runs/3fcde2c8-b461-4f22-afec-daa269291c95/dangling' of persistent volume '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/volumes/roles/default-role/persistence-id' inside garbage collected path '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000/executors/test-task123'
      W0702 23:35:31.683878 25401 gc.cpp:241] Unmounting dangling mount point '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000/executors/test-task123/runs/3fcde2c8-b461-4f22-afec-daa269291c95/dangling' of persistent volume '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/volumes/roles/default-role/persistence-id' inside garbage collected path '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000'
      W0702 23:35:31.683912 25401 gc.cpp:248] Skipping deletion of '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000' because unmount failed on '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000/executors/test-task123/runs/3fcde2c8-b461-4f22-afec-daa269291c95/dangling': Failed to unmount '/tmp/GarbageCollectorIntegrationTest_ROOT_DanglingMount_zkItvU/slaves/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-S0/frameworks/f4dc0941-e3b0-4f2c-b7f9-025a1af264c8-0000/executors/test-task123/runs/3fcde2c8-b461-4f22-afec-daa269291c95/dangling': Invalid argument
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zhitao Zhitao Li
                Reporter:
                chhsia0 Chun-Hung Hsiao
                Shepherd:
                Chun-Hung Hsiao
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: