Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21377

Jars specified with --jars or --packages are not added into AM's system classpath

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: YARN
    • Labels:
      None

      Description

      In this issue we have a long running Spark application with secure HBase, which requires HBaseCredentialProvider to get tokens periodically, we specify HBase related jars with --packages, but these dependencies are not added into AM classpath, so when HBaseCredentialProvider tries to initialize HBase connections to get tokens, it will be failed.

      Currently because jars specified with --jars or --packages are not added into AM classpath, the only way to extend AM classpath is to use "spark.driver.extraClassPath" which supposed to be used in yarn cluster mode.

      So here we should figure out a solution either to put these dependencies to AM classpath or to extend AM classpath with correct configuration.

        Attachments

          Activity

            People

            • Assignee:
              jerryshao Saisai Shao
              Reporter:
              yeshavora Yesha Vora
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: