Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36493

Skip Retrieving keytab with SparkFiles.get if keytab found in the CWD of Yarn Container

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.0, 3.1.2
    • Fix Version/s: 3.1.3
    • Component/s: Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      Currently we have the logic to deal with the JDBC keytab provided by the "--files" option

      if (keytabParam != null && FilenameUtils.getPath(keytabParam).isEmpty)
      {{{}}
      }}{{val result = SparkFiles.get(keytabParam)
      }}{{logDebug(s"Keytab path not found, assuming --files, file name used on executor: $result")
      }}{{result
      } else {
      }}{{logDebug("Keytab path found, assuming manual upload")
      }}{{keytabParam
      }

      Spark has already created the soft link for any file submitted by the "--files" option. Here is an example.

      testusera1.keytab -> /var/opt/hadoop/temp/nm-local-dir/usercache/testusera1/appcache/application_1628584679772_0003/filecache/12/testusera1.keytab

       

      So there is no need to call the SparkFiles.get to absolute path of the keytab file. We can directly use the variable `keytabParam` as the keytab file path.

       

      Moreover, SparkFiles.get will get a wrong path of keytab for the driver in cluster mode. In cluster mode, the keytab is available at the following location for both the driver and executors

      /var/opt/hadoop/temp/nm-local-dir/usercache/testusera1/appcache/application_1628584679772_0003/container_1628584679772_0030_01_000001/testusera1.keytab

      but SparkFiles.get brings the following wrong location for the driver

      /var/opt/hadoop/temp/nm-local-dir/usercache/testusera1/appcache/application_1628584679772_0003/spark-8fb0f437-c842-4a9f-9612-39de40082e40/userFiles-5075388b-0928-4bc3-a498-7f6c84b27808/testusera1.keytab

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              xuzikun2003 Zikun
              Shepherd:
              Gabor Somogyi

              Dates

              • Created:
                Updated:

                Issue deployment