Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7281

No option for AM native library path in yarn-client mode.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.3.1, 1.4.0
    • Fix Version/s: 1.4.0
    • Component/s: YARN
    • Labels:
      None

      Description

      There are many options to control the AM in yarn-client mode, but the option for the native library path is missing. Without that, users have to change the process's environment directly (using spark.yarn.appMasterEnv.LD_LIBRARY_PATH on Linux, for example) to add things to the native library path.

      Depending on the cluster's configuration, native libraries are needed in the AM (for example when using the JNI-based group mapper in Hadoop).

      There's also no option to control the AM's classpath, but since the AM is not running user code, that seems less important.

        Attachments

          Activity

            People

            • Assignee:
              vanzin Marcelo Masiero Vanzin
              Reporter:
              vanzin Marcelo Masiero Vanzin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: