[SPARK-4612] Configuration object gets created for every task even if not new file/jar is added

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Target Version/s: 1.2.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 26/Nov/14 01:42
• Updated: 26/Nov/14 15:16
• Resolved: 26/Nov/14 15:16

Description

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/Executor.scala#L337 creates a configuration object for every task that is launched, even if there is no new dependent file/JAR to update. This is a heavy-weight creation that should be avoided if there is not new file/JAR to update.

Issue Links

• links to: [Github] Pull Request #3463 (tdas)