[SPARK-31551] createSparkUser lost user's non-Hadoop credentials

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.4, 2.4.5
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Yuqi Wang
• Created: 24/Apr/20 05:38
• Updated: 25/May/21 01:53
• Resolved: 25/May/21 01:44

Description

See current createSparkUser:

   def createSparkUser(): UserGroupInformation = {
    val user = Utils.getCurrentUserName()
    logDebug("creating UGI for user: " + user)
    val ugi = UserGroupInformation.createRemoteUser(user)
    transferCredentials(UserGroupInformation.getCurrentUser(), ugi)
    ugi
  }

  def transferCredentials(source: UserGroupInformation, dest: UserGroupInformation): Unit = {
    dest.addCredentials(source.getCredentials())
  }

  def getCurrentUserName(): String = {
    Option(System.getenv("SPARK_USER"))
      .getOrElse(UserGroupInformation.getCurrentUser().getShortUserName())
  }

The transferCredentials func can only transfer Hadoop creds such as Delegation Tokens.
However, other creds stored in UGI.subject.getPrivateCredentials, will be lost here, such as:

1. Non-Hadoop creds:
   Such as, Kafka creds
2. Newly supported or 3rd party supported Hadoop creds:
   Such as to support OAuth/JWT token authn on Hadoop, we need to store the OAuth/JWT token into UGI.subject.getPrivateCredentials. However, these tokens are not supposed to be managed by Hadoop Credentials (currently it is only for Hadoop secret keys and delegation tokens)

Another issue is that the SPARK_USER only gets the UserGroupInformation.getCurrentUser().getShortUserName() of the user, which may lost the user's fully qualified user name. We should better use the getUserName to get fully qualified user name in our client side, which is aligned to HADOOP_PROXY_USER.

Related to https://issues.apache.org/jira/browse/SPARK-1051

Issue Links

• is caused by: SPARK-1051 On Yarn, executors don't doAs as submitting user
• links to: GitHub Pull Request #28323