[SPARK-2713] Executors of same application in same host should only download files & jars once

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Target Version/s: 1.2.0
• Assignee: Zhihui
• Reporter: Zhihui
• Created: 28/Jul/14 05:25
• Updated: 31/Mar/15 01:21
• Resolved: 24/Oct/14 20:02

Description

If Spark lunched multiple executors in one host for one application, every executor would download it dependent files and jars (if not using local: url) independently. It maybe result in huge latency. In my case, it result in 20 seconds latency to download dependent jars(about 17M) when I lunch 32 executors in one host(total 4 hosts).

This patch will cache downloaded files and jars for executors to reduce network throughput and download latency. I my case, the latency was reduced from 20 seconds to less than 1 second.

Issue Links

• is related to: SPARK-662 Executor should only download files & jars once
• relates to: SPARK-6619 Improve Jar caching on executors
• links to: [Github] Pull Request #1616 (li-zhihui)