[MESOS-8280] Mesos Containerizer GC should set 'layers' after checkpointing layer ids in provisioner. - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.5.0
Component/s: image-gc, provisioner
Labels:

Target Version/s:

1.5.0
Epic Link:
UCR GC
Sprint:
Mesosphere Sprint 69
Story Points:
3

Description

11111
222222
333333
444444
11111
222222
333333
444444
I1129 23:24:45.469543  6592 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/MVgVC7/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/MVgVC7/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e/rootfs.overlay'
I1129 23:24:45.473287  6592 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/MVgVC7/sha256:b56ae66c29370df48e7377c8f9baa744a3958058a766793f821dadcb144a4647 to rootfs '/tmp/mesos/store/docker/staging/MVgVC7/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3/rootfs.overlay'
I1129 23:24:45.582002  6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6/rootfs.overlay'
I1129 23:24:45.589404  6595 metadata_manager.cpp:167] Successfully cached image 'alpine'
I1129 23:24:45.590204  6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e/rootfs.overlay'
I1129 23:24:45.595190  6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f/rootfs.overlay'
I1129 23:24:45.599500  6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721/rootfs.overlay'
I1129 23:24:45.602047  6597 provisioner.cpp:506] Provisioning image rootfs '/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/rootfses/b5d48445-848d-4274-a4f8-e909351ebc35' for container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 using overlay backend
I1129 23:24:45.602751  6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:1db09adb5ddd7f1a07b6d585a7db747a51c7bd17418d47e91f901bdf420abd66 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16/rootfs.overlay'
I1129 23:24:45.603054  6596 overlay.cpp:168] Created symlink '/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/scratch/b5d48445-848d-4274-a4f8-e909351ebc35/links' -> '/tmp/xAWQ8y'
I1129 23:24:45.604398  6596 overlay.cpp:196] Provisioning image rootfs with overlayfs: 'lowerdir=/tmp/xAWQ8y/1:/tmp/xAWQ8y/0,upperdir=/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/scratch/b5d48445-848d-4274-a4f8-e909351ebc35/upperdir,workdir=/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/scratch/b5d48445-848d-4274-a4f8-e909351ebc35/workdir'
I1129 23:24:45.607802  6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229/rootfs.overlay'
I1129 23:24:45.612139  6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/rootfs.overlay'
I1129 23:24:45.612253  6593 containerizer.cpp:1369] Checkpointed ContainerConfig at '/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/config'
I1129 23:24:45.612298  6593 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from PROVISIONING to PREPARING
I1129 23:24:45.625658  6596 containerizer.cpp:1838] Launching 'mesos-containerizer' with flags '--help="false" --launch_info="{"clone_namespaces":[131072],"command":{"shell":true,"value":"sleep 1"},"environment":{"variables":[{"name":"MESOS_SANDBOX","type":"VALUE","value":"\/mnt\/mesos\/sandbox"},{"name":"PATH","type":"VALUE","value":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"},{"name":"MESOS_CONTAINER_IP","type":"VALUE","value":"10.0.2.15"}]},"pre_exec_commands":[{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/vagrant\/mesos\/build\/src\/mesos-containerizer"},{"arguments":["mount","-n","--rbind","\/tmp\/slaves\/65fa58c5-48c6-4998-b336-ceb9bcd2ec43-S0\/frameworks\/55b825d4-c922-46bb-ab8e-4e01abf1a756-0000\/executors\/default-executor\/runs\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/01de09c5-d8e9-412e-8825-a592d2c875e5","\/tmp\/provisioner\/containers\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/01de09c5-d8e9-412e-8825-a592d2c875e5\/backends\/overlay\/rootfses\/b5d48445-848d-4274-a4f8-e909351ebc35\/mnt\/mesos\/sandbox"],"shell":false,"value":"mount"}],"rootfs":"\/tmp\/provisioner\/containers\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/01de09c5-d8e9-412e-8825-a592d2c875e5\/backends\/overlay\/rootfses\/b5d48445-848d-4274-a4f8-e909351ebc35","task_environment":{},"user":"root","working_directory":"\/mnt\/mesos\/sandbox"}" --pipe_read="12" --pipe_write="15" --runtime_directory="/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5" --unshare_namespace_mnt="false"'
I1129 23:24:45.626317  6598 linux_launcher.cpp:438] Launching nested container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 and cloning with namespaces CLONE_NEWNS
I1129 23:24:45.633211  6598 systemd.cpp:96] Assigned child process '6745' to 'mesos_executors.slice'
I1129 23:24:45.636270  6596 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from PREPARING to ISOLATING
I1129 23:24:45.691830  6597 metadata_manager.cpp:167] Successfully cached image 'mesosphere/inky'
I1129 23:24:45.694399  6594 provisioner.cpp:506] Provisioning image rootfs '/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/backends/overlay/rootfses/1187cc83-a23a-4390-9c28-092a7b7690b5' for container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 using overlay backend
I1129 23:24:45.694919  6596 overlay.cpp:168] Created symlink '/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/backends/overlay/scratch/1187cc83-a23a-4390-9c28-092a7b7690b5/links' -> '/tmp/GXhXiT'
I1129 23:24:45.695103  6596 overlay.cpp:196] Provisioning image rootfs with overlayfs: 'lowerdir=/tmp/GXhXiT/6:/tmp/GXhXiT/5:/tmp/GXhXiT/4:/tmp/GXhXiT/3:/tmp/GXhXiT/2:/tmp/GXhXiT/1:/tmp/GXhXiT/0,upperdir=/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/backends/overlay/scratch/1187cc83-a23a-4390-9c28-092a7b7690b5/upperdir,workdir=/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/backends/overlay/scratch/1187cc83-a23a-4390-9c28-092a7b7690b5/workdir'
I1129 23:24:45.696255  6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers
I1129 23:24:45.696349  6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers
I1129 23:24:45.696449  6593 containerizer.cpp:1369] Checkpointed ContainerConfig at '/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/config'
I1129 23:24:45.696506  6593 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 from PROVISIONING to PREPARING
I1129 23:24:45.697865  6595 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache
I1129 23:24:45.697918  6595 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache
I1129 23:24:45.697968  6595 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache
I1129 23:24:45.697999  6595 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache
I1129 23:24:45.698025  6595 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache
I1129 23:24:45.698050  6595 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache
I1129 23:24:45.698076  6595 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache
I1129 23:24:45.698104  6595 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache
I1129 23:24:45.698129  6595 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache
I1129 23:24:45.698894  6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers
I1129 23:24:45.698966  6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers
I1129 23:24:45.700333  6596 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache
I1129 23:24:45.700394  6596 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache
I1129 23:24:45.700412  6596 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache
I1129 23:24:45.700428  6596 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache
I1129 23:24:45.700441  6596 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache
I1129 23:24:45.700454  6596 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache
I1129 23:24:45.700467  6596 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache
I1129 23:24:45.700480  6596 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache
I1129 23:24:45.700495  6596 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache
I1129 23:24:45.702491  6594 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers
I1129 23:24:45.702554  6594 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers
I1129 23:24:45.703707  6592 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache
I1129 23:24:45.703783  6592 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache
I1129 23:24:45.703812  6592 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache
I1129 23:24:45.703816  6592 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache
I1129 23:24:45.703816  6592 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache
I1129 23:24:45.704164  6592 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache
I1129 23:24:45.704208  6592 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache
I1129 23:24:45.704255  6592 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache
I1129 23:24:45.704285  6592 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache
I1129 23:24:45.704814  6595 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers
I1129 23:24:45.704861  6595 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers
I1129 23:24:45.708112  6592 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache
I1129 23:24:45.708204  6592 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache
I1129 23:24:45.708238  6592 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache
I1129 23:24:45.708264  6592 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache
I1129 23:24:45.708375  6592 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache
I1129 23:24:45.708407  6592 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache
I1129 23:24:45.708472  6592 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache
I1129 23:24:45.708514  6592 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache
I1129 23:24:45.708545  6592 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache
I1129 23:24:45.709048  6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers
I1129 23:24:45.709161  6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers
I1129 23:24:45.710321  6594 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache
I1129 23:24:45.710382  6594 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache
I1129 23:24:45.710412  6594 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache
I1129 23:24:45.710436  6594 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache
I1129 23:24:45.710458  6594 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache
I1129 23:24:45.710480  6594 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache
I1129 23:24:45.710522  6594 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache
I1129 23:24:45.710551  6594 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache
I1129 23:24:45.710590  6594 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache
I1129 23:24:45.713176  6594 containerizer.cpp:1838] Launching 'mesos-containerizer' with flags '--help="false" --launch_info="{"clone_namespaces":[131072],"command":{"shell":true,"value":"sleep 100000"},"environment":{"variables":[{"name":"MESOS_SANDBOX","type":"VALUE","value":"\/mnt\/mesos\/sandbox"},{"name":"HOME","type":"VALUE","value":"\/"},{"name":"PATH","type":"VALUE","value":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"},{"name":"MESOS_CONTAINER_IP","type":"VALUE","value":"10.0.2.15"}]},"pre_exec_commands":[{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/vagrant\/mesos\/build\/src\/mesos-containerizer"},{"arguments":["mount","-n","--rbind","\/tmp\/slaves\/65fa58c5-48c6-4998-b336-ceb9bcd2ec43-S0\/frameworks\/55b825d4-c922-46bb-ab8e-4e01abf1a756-0000\/executors\/default-executor\/runs\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/7be37efd-7ec4-459a-a51a-97f6be99c3c3","\/tmp\/provisioner\/containers\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/7be37efd-7ec4-459a-a51a-97f6be99c3c3\/backends\/overlay\/rootfses\/1187cc83-a23a-4390-9c28-092a7b7690b5\/mnt\/mesos\/sandbox"],"shell":false,"value":"mount"}],"rootfs":"\/tmp\/provisioner\/containers\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/7be37efd-7ec4-459a-a51a-97f6be99c3c3\/backends\/overlay\/rootfses\/1187cc83-a23a-4390-9c28-092a7b7690b5","task_environment":{},"user":"root","working_directory":"\/mnt\/mesos\/sandbox"}" --pipe_read="13" --pipe_write="14" --runtime_directory="/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3" --unshare_namespace_mnt="false"'
I1129 23:24:45.713954  6597 linux_launcher.cpp:438] Launching nested container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 and cloning with namespaces CLONE_NEWNS
I1129 23:24:45.721781  6597 systemd.cpp:96] Assigned child process '6775' to 'mesos_executors.slice'
I1129 23:24:45.725494  6594 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 from PREPARING to ISOLATING
I1129 23:24:45.791635  6595 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from ISOLATING to FETCHING
I1129 23:24:45.791880  6591 fetcher.cpp:379] Starting to fetch URIs for container: 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5, directory: /tmp/slaves/65fa58c5-48c6-4998-b336-ceb9bcd2ec43-S0/frameworks/55b825d4-c922-46bb-ab8e-4e01abf1a756-0000/executors/default-executor/runs/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5
I1129 23:24:45.792626  6591 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from FETCHING to RUNNING
11111
222222
333333
444444
I1129 23:24:45.807262  6592 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers
I1129 23:24:45.807375  6592 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers
I1129 23:24:45.808658  6591 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache
I1129 23:24:45.808843  6591 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache
I1129 23:24:45.808869  6591 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache
I1129 23:24:45.808897  6591 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache
I1129 23:24:45.808962  6591 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache
I1129 23:24:45.808990  6591 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache
I1129 23:24:45.809012  6591 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache
I1129 23:24:45.809036  6591 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache
I1129 23:24:45.809057  6591 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache
I1129 23:24:45.893280  6596 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 from ISOLATING to FETCHING
I1129 23:24:45.893523  6596 fetcher.cpp:379] Starting to fetch URIs for container: 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3, directory: /tmp/slaves/65fa58c5-48c6-4998-b336-ceb9bcd2ec43-S0/frameworks/55b825d4-c922-46bb-ab8e-4e01abf1a756-0000/executors/default-executor/runs/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3
I1129 23:24:45.894335  6594 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 from FETCHING to RUNNING
I1129 23:24:45.902606  6598 process.cpp:3503] Handling HTTP event for process 'slave(1)' with path: '/slave(1)/api/v1/executor'
I1129 23:24:45.903908  6597 process.cpp:3503] Handling HTTP event for process 'slave(1)' with path: '/slave(1)/api/v1'
I1129 23:24:45.904618  6597 process.cpp:3503] Handling HTTP event for process 'slave(1)' with path: '/slave(1)/api/v1'
I1129 23:24:45.908681  6597 http.cpp:1185] HTTP POST for /slave(1)/api/v1 from 10.0.2.15:57620
I1129 23:24:45.909113  6597 http.cpp:1185] HTTP POST for /slave(1)/api/v1 from 10.0.2.15:57622
I1129 23:24:45.909708  6597 http.cpp:2589] Processing WAIT_NESTED_CONTAINER call for container '3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5'
I1129 23:24:45.910148  6597 http.cpp:2589] Processing WAIT_NESTED_CONTAINER call for container '3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3'
I1129 23:24:45.938350  6596 process.cpp:3503] Handling HTTP event for process 'slave(1)' with path: '/slave(1)/api/v1/executor'
I1129 23:24:45.938781  6596 http.cpp:1185] HTTP POST for /slave(1)/api/v1/executor from 10.0.2.15:57564
I1129 23:24:45.939141  6596 slave.cpp:4584] Handling status update TASK_RUNNING (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.940809  6591 http.cpp:1185] HTTP POST for /slave(1)/api/v1/executor from 10.0.2.15:57564
I1129 23:24:45.941130  6591 slave.cpp:4584] Handling status update TASK_RUNNING (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.942906  6591 task_status_update_manager.cpp:328] Received task status update TASK_RUNNING (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.943076  6591 task_status_update_manager.cpp:383] Forwarding task status update TASK_RUNNING (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to the agent
I1129 23:24:45.943322  6595 slave.cpp:5067] Forwarding the update TASK_RUNNING (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to master@127.0.0.1:5050
I1129 23:24:45.943332  6591 task_status_update_manager.cpp:328] Received task status update TASK_RUNNING (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.943506  6591 task_status_update_manager.cpp:383] Forwarding task status update TASK_RUNNING (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to the agent
I1129 23:24:45.943717  6595 slave.cpp:4960] Task status update manager successfully handled status update TASK_RUNNING (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.943905  6595 slave.cpp:5067] Forwarding the update TASK_RUNNING (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to master@127.0.0.1:5050
I1129 23:24:45.943905  6595 slave.cpp:4960] Task status update manager successfully handled status update TASK_RUNNING (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.992866  6591 task_status_update_manager.cpp:401] Received task status update acknowledgement (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.993261  6595 slave.cpp:3868] Task status update manager successfully handled status update acknowledgement (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.993968  6593 task_status_update_manager.cpp:401] Received task status update acknowledgement (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:45.994295  6598 slave.cpp:3868] Task status update manager successfully handled status update acknowledgement (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
11111
222222
333333
444444
I1129 23:24:46.808684  6592 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers
I1129 23:24:46.808876  6592 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers
I1129 23:24:46.810683  6593 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache
I1129 23:24:46.810751  6593 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache
I1129 23:24:46.810781  6593 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache
I1129 23:24:46.810808  6593 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache
I1129 23:24:46.810834  6593 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache
I1129 23:24:46.810860  6593 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache
I1129 23:24:46.810885  6593 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache
I1129 23:24:46.810911  6593 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache
I1129 23:24:46.810937  6593 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache
I1129 23:24:47.057590  6596 containerizer.cpp:2775] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has exited
I1129 23:24:47.057667  6596 containerizer.cpp:2324] Destroying container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 in RUNNING state
I1129 23:24:47.057695  6596 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from RUNNING to DESTROYING
I1129 23:24:47.058082  6596 linux_launcher.cpp:514] Asked to destroy container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5
I1129 23:24:47.059027  6596 linux_launcher.cpp:560] Using freezer to destroy cgroup mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5
I1129 23:24:47.060667  6596 cgroups.cpp:3058] Freezing cgroup /sys/fs/cgroup/freezer/mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5
I1129 23:24:47.062700  6597 cgroups.cpp:1413] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5 after 1.838336ms
I1129 23:24:47.064627  6592 cgroups.cpp:3076] Thawing cgroup /sys/fs/cgroup/freezer/mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5
I1129 23:24:47.066498  6598 cgroups.cpp:1442] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5 after 1.642752ms
I1129 23:24:47.071521  6592 provisioner.cpp:648] Destroying container rootfs at '/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/rootfses/b5d48445-848d-4274-a4f8-e909351ebc35' for container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5
I1129 23:24:47.098203  6596 overlay.cpp:296] Removed temporary directory '/tmp/xAWQ8y' pointed by '/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/scratch/b5d48445-848d-4274-a4f8-e909351ebc35/links'
I1129 23:24:47.100265  6591 containerizer.cpp:2613] Checkpointing termination state to nested container's runtime directory '/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/termination'
I1129 23:24:47.107206  6594 process.cpp:3503] Handling HTTP event for process 'slave(1)' with path: '/slave(1)/api/v1/executor'
I1129 23:24:47.151911  6594 http.cpp:1185] HTTP POST for /slave(1)/api/v1/executor from 10.0.2.15:57564
I1129 23:24:47.152243  6594 slave.cpp:4584] Handling status update TASK_FINISHED (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:47.154391  6594 task_status_update_manager.cpp:328] Received task status update TASK_FINISHED (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:47.154578  6594 task_status_update_manager.cpp:383] Forwarding task status update TASK_FINISHED (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to the agent
I1129 23:24:47.154810  6593 slave.cpp:5067] Forwarding the update TASK_FINISHED (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to master@127.0.0.1:5050
I1129 23:24:47.157249  6593 slave.cpp:4960] Task status update manager successfully handled status update TASK_FINISHED (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:47.385977  6592 task_status_update_manager.cpp:401] Received task status update acknowledgement (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:47.386334  6592 task_status_update_manager.cpp:538] Cleaning up status update stream for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:47.387459  6592 slave.cpp:3868] Task status update manager successfully handled status update acknowledgement (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000
I1129 23:24:47.387568  6592 slave.cpp:8457] Completing task 1
11111
222222
333333
444444
I1129 23:24:47.818768  6595 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers
I1129 23:24:47.824591  6598 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache
I1129 23:24:47.824724  6598 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache
I1129 23:24:47.824753  6598 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache
I1129 23:24:47.824767  6598 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache
I1129 23:24:47.824782  6598 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache
I1129 23:24:47.824861  6598 store.cpp:550] Marking layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' to gc by renaming '/tmp/mesos/store/docker/layers/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' to '/tmp/mesos/store/docker/gc/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e.1511997887824822784'
I1129 23:24:47.824918  6598 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache
I1129 23:24:47.824960  6598 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache
I1129 23:24:47.825088  6598 store.cpp:550] Marking layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' to gc by renaming '/tmp/mesos/store/docker/layers/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' to '/tmp/mesos/store/docker/gc/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3.1511997887825015040'
I1129 23:24:47.825389  6598 store.cpp:577] Deleting path '/tmp/mesos/store/docker/gc/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3.1511997887825015040'
I1129 23:24:47.829903  6598 store.cpp:584] Deleted '/tmp/mesos/store/docker/gc/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3.1511997887825015040'
I1129 23:24:47.829980  6598 store.cpp:577] Deleting path '/tmp/mesos/store/docker/gc/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e.1511997887824822784'
I1129 23:24:47.830047  6598 store.cpp:584] Deleted '/tmp/mesos/store/docker/gc/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e.1511997887824822784'
11111
222222
333333
444444
I1129 23:24:48.829519  6595 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers
I1129 23:24:48.831161  6598 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache
I1129 23:24:48.831225  6598 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache
I1129 23:24:48.831248  6598 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache
I1129 23:24:48.831266  6598 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache
I1129 23:24:48.831284  6598 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache
I1129 23:24:48.831302  6598 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache
I1129 23:24:48.831321  6598 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache
11111
222222
333333
444444
I1129 23:24:49.830904  6597 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers
I1129 23:24:49.832487  6597 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache
I1129 23:24:49.832584  6597 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache
I1129 23:24:49.833329  6597 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache
I1129 23:24:49.833367  6597 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache
I1129 23:24:49.833387  6597 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache
I1129 23:24:49.833406  6597 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache
I1129 23:24:49.833425  6597 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache

Please neglect the debugging logs like '111111'. To reproduce this issue, just continuously trigger image gc. The log above was from a scenario that we launch two nested containers. One sleeps 1 second, another sleep forever.

This is related to this patch: https://github.com/apache/mesos/commit/e273efe6976434858edb85bbcf367a02e963a467#diff-a3593ed0ebd2b205775f7f04d9b5afe7

The root cause is that we did not set the 'layers' after we checkpoint the layer ids in provisioner. The log below is the prove:

I1129 23:24:45.698894  6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers
I1129 23:24:45.698966  6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers

Attachments

Activity

People

Assignee:: Zhitao Li

Reporter:: Gilbert Song

Shepherd:: Gilbert Song

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 30/Nov/17 00:04

Updated:: 07/Dec/17 16:56

Resolved:: 07/Dec/17 16:56