[MESOS-9221] If some image layers are large, the image pulling may stuck due to the authorized token expired.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.4.2, 1.5.0
• Component/s: containerization
• Labels: containerizer
• Target Version/s: 1.4.3, 1.5.2
• Epic Link: Unified Container
• Story Points: 5
• Assignee: Chun-Hung Hsiao
• Reporter: Gilbert Song
• Created: 08/Sep/18 01:43
• Updated: 27/Oct/18 00:13
• Resolved: 27/Oct/18 00:11

Description

The image layer blobs pulling happen asynchronously but in the same libprocess process. There is a chance that one layer get the token then the thread switch to another layer curling which may take long. When the original layer curling resumes, the token already expired (e.g., after 60 seconds).

$ sudo cat /var/lib/mesos/slave/store/docker/staging/0gx64f/sha256\:c75480ad9aafadef6c7faf829ede40cf2fa990c9308d6cd354d53041b01a7cda
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"mesosphere/dapis","Action":"pull"}]}]}

The impact is the task launch stuck and all subsequent task using this image would also stuck because it waits for the same image pulling future to become ready.

Please note that this issue is not likely to be reproduced, unless on a busy system using images containing large layers.

Issue Links

• duplicates: MESOS-9320 UCR container launch stuck at PROVISIONING during image fetching.
• is related to: MESOS-8620 Containers stuck in FETCHING possibly due to unresponsive server.