[SPARK-15994] Allow enabling Mesos fetch cache in coarse executor backend

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.1.0
• Component/s: Mesos
• Assignee: Charles R Allen
• Reporter: Charles R Allen
• Created: 16/Jun/16 17:48
• Updated: 07/Feb/19 09:12
• Resolved: 01/Nov/16 13:14

Description

Mesos 0.23.0 introduces a Fetch Cache feature http://mesos.apache.org/documentation/latest/fetcher/ which allows caching of resources specified in command URIs.

This patch:

• Updates the Mesos shaded protobuf dependency to 0.23.0
• Allows setting `spark.mesos.fetchCache.enable` to enable the fetch cache for all specified URIs. (URIs must be specified for the setting to have any affect)
• Updates documentation for Mesos configuration with the new setting.

This patch does NOT:

• Allow for per-URI caching configuration. The cache setting is global to ALL URIs for the command.

Issue Links

• causes: SPARK-26082 Misnaming of spark.mesos.fetch(er)Cache.enable in MesosClusterScheduler
• relates to: SPARK-11183 enable support for mesos 0.24+
• links to: [Github] Pull Request #13713 (drcrallen)