[SPARK-8619] Can't find the keytab file when recovering the streaming application.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.1, 1.5.0
• Component/s: DStreams
• Target Version/s: 1.4.1, 1.4.2
• Assignee: Zhaowei Huang
• Reporter: Zhaowei Huang
• Created: 25/Jun/15 03:32
• Updated: 04/Jul/15 07:27
• Resolved: 30/Jun/15 18:46

Description

In a streaming application, I use --keytab /root/spark.keytab to get the token.
But when the streaming application failed and I wanted to recover it from checkpoint file, there was an error:

java.io.IOException: Login failure for spark/[e-mail] from keytab spark.keytab-1fd8f7bb-0d3c-4f65-990a-9ae09055cc8d: javax.security.auth.login.LoginException: Unable to obtain password from user

Spark had changed the configuration, so the checkpoint can't find the file:

Client.java @ Function: setupCredentials

      val keytabFileName = f.getName + "-" + UUID.randomUUID().toString
      UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab)
      loginFromKeytab = true
      sparkConf.set("spark.yarn.keytab", keytabFileName)

So when recovering the application, we should ignore this configurations.

Issue Links

• links to: [Github] Pull Request #7008 (SaintBacchus)