[SPARK-21377] Jars specified with --jars or --packages are not added into AM's system classpath

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Spark Core, YARN
• Assignee: Saisai Shao
• Reporter: Yesha Vora
• Created: 11/Jul/17 18:48
• Updated: 17/May/20 18:14
• Resolved: 17/Jul/17 20:12

Description

In this issue we have a long running Spark application with secure HBase, which requires HBaseCredentialProvider to get tokens periodically, we specify HBase related jars with --packages, but these dependencies are not added into AM classpath, so when HBaseCredentialProvider tries to initialize HBase connections to get tokens, it will be failed.

Currently because jars specified with --jars or --packages are not added into AM classpath, the only way to extend AM classpath is to use "spark.driver.extraClassPath" which supposed to be used in yarn cluster mode.

So here we should figure out a solution either to put these dependencies to AM classpath or to extend AM classpath with correct configuration.

Issue Links

• links to: [Github] Pull Request #18602 (jerryshao)
• links to: [Github] Pull Request #18616 (jerryshao)