[SPARK-36493] Skip Retrieving keytab with SparkFiles.get if keytab found in the CWD of Yarn Container

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0, 3.1.2
• Fix Version/s: 3.1.3
• Component/s: Spark Core
• Target Version/s: 3.1.2
• Assignee: Unassigned
• Reporter: Zikun
• Shepherd: Gabor Somogyi
• Created: 12/Aug/21 10:09
• Updated: 17/Aug/21 22:15

Description

Currently we have the logic to deal with the JDBC keytab provided by the "--files" option

if (keytabParam != null && FilenameUtils.getPath(keytabParam).isEmpty)
{{{}}
}}{{val result = SparkFiles.get(keytabParam)
}}{{logDebug(s"Keytab path not found, assuming --files, file name used on executor: $result")
}}{{result
} else {
}}{{logDebug("Keytab path found, assuming manual upload")
}}{{keytabParam
}

Spark has already created the soft link for any file submitted by the "--files" option. Here is an example.

testusera1.keytab -> /var/opt/hadoop/temp/nm-local-dir/usercache/testusera1/appcache/application_1628584679772_0003/filecache/12/testusera1.keytab

 

So there is no need to call the SparkFiles.get to absolute path of the keytab file. We can directly use the variable `keytabParam` as the keytab file path.

 

Moreover, SparkFiles.get will get a wrong path of keytab for the driver in cluster mode. In cluster mode, the keytab is available at the following location for both the driver and executors

/var/opt/hadoop/temp/nm-local-dir/usercache/testusera1/appcache/application_1628584679772_0003/container_1628584679772_0030_01_000001/testusera1.keytab

but SparkFiles.get brings the following wrong location for the driver

/var/opt/hadoop/temp/nm-local-dir/usercache/testusera1/appcache/application_1628584679772_0003/spark-8fb0f437-c842-4a9f-9612-39de40082e40/userFiles-5075388b-0928-4bc3-a498-7f6c84b27808/testusera1.keytab

 

 

Issue Links

• links to: https://github.com/apache/spark/pull/33726