Major Description
What happens here is that we are erroneously applying the sandbox's project ID to the persistent volume.
First, the filesystem/linux isolator bind mounts the persistent volume into the sandbox:
I1003 06:49:21.907644 2812466 linux.cpp:593] Mounting '/srv/mesos/work/volumes/roles/pie.mobius/21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f' to '/srv/mesos/work/slaves/909cff92-8e17-41bf-a251-9b5eb6186c35-S0/frameworks/363e6d80-8c38-46cf-815f-2fbf60a62628-0309/executors/mobius-mloop-1538549013_438156792-v2-shared-volume.pod1.writer-job.0.e93hs3uips2i9_1/runs/9e5770a7-9f78-46dc-9264-3e80be0e40cc/shared' for persistent volume disk(allocated: pie.mobius)(reservations: [(DYNAMIC,pie.mobius,jarvis-principal,\{podInstance: e93hs3uips2i9, pod: pod1, service: mobius-mloop-1538549013_438156792-v2-shared-volume})])[21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f:shared]<SHARED>:1 of container 9e5770a7-9f78-46dc-9264-3e80be0e40cc
Next, the `disk/xfs` isolator assigns a project ID to the sandbox:
I1003 06:49:21.920197 2812452 disk.cpp:402] Assigned project 6806 to '/srv/mesos/work/slaves/909cff92-8e17-41bf-a251-9b5eb6186c35-S0/frameworks/363e6d80-8c38-46cf-815f-2fbf60a62628-0309/executors/mobius-mloop-1538549013_438156792-v2-shared-volume.pod1.writer-job.0.e93hs3uips2i9_1/runs/9e5770a7-9f78-46dc-9264-3e80be0e40cc'
Note, that when this happens, the isolator recursively applies the project ID to the contents of the sandbox. It doesn't follow symlinks or cross devices when it does this, but on Linux, a bind mount would not trigger either of these conditions.
Finally, the `disk/xfs` isolator tries to assign a project ID to the persistent volume as it is used by the task:
F1003 06:49:21.920577 2812452 disk.cpp:532] Check failed: scheduledProjects.contains(projectId.get()) untracked project ID 6806 for volume ID 21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f on /srv/mesos/work/volumes/roles/pie.mobius/21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f
This check fails, because if the persistent volume has a project ID, we expect that is had already be scheduled for reclaimation. However, it's project ID is the one we assigned to the sandbox. We don't scheduled the ssandbox for reclaimation until cleanup, so (fortunately) the invariant check triggers.
So, apart from triggering the CHECK, the root cause of this is that we are altering the project ID of the persistent volume, which permanently misattributes the corresponding quote.