[MESOS-3918] Unified and pluggable URI fetching support.

• Type: Epic
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Labels: mesosphere twitter
• Epic Name: URI Fetcher
• Assignee: Jie Yu
• Reporter: Jie Yu
• Created: 13/Nov/15 18:57
• Updated: 29/Jul/16 20:59

Description

Fetcher was originally designed to fetch CommandInfo::URIs (e.g., executor binary) for executors/tasks. A recent refactor (MESOS-336) added caching support to the fetcher. The recent work on filesystem isolation/unified containerizer (MESOS-2840) requires Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The natural question is: can we leverage the fetcher to fetch those filesystem images (and cache them accordingly)? Unfortunately, the existing fetcher interface is tightly coupled with CommandInfo::URIs for executors/tasks, making it very hard to be re used to fetch/cache filesystem images.

Another motivation for the refactor is that we want to extend the fetcher to support more types of schemes. For instance, we want to support magnet URI to enable p2p fetching. This is in fact quite important for operating a large cluster (MESOS-3596). The goal here is to allow fetcher to be extended (e.g., using modules) so that operators can add custom fetching support.

The main design goal here is to decouple artifacts fetching from artifacts cache management. We can make artifacts fetching extensible (e.g. to support p2p fetching), and solve the cache management part later.

Issue Links

• relates to: MESOS-2731 Allow frameworks to deploy storage drivers on demand.
• relates to: MESOS-3511 Generalize fetcher to handle container image fetching
• relates to: MESOS-4296 Add docker URI fetcher plugin based on curl.