[AMBARI-25311] FinalizeKerberosServerAction timeout has to be configurable

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.3
• Fix Version/s: 2.8.0
• Component/s: ambari-server
• Labels: pull-request-available
• Assignee: amarnath reddy pappu
• Reporter: amarnath reddy pappu
• Created: 17/Jun/19 21:22
• Updated: 25/Oct/22 16:10
• Resolved: 09/Jul/19 20:22

Description

During the HDP Upgrade to 3.1 there is re-generate keytab operation which is timing out in some cases. here is the example log.

2019-06-11 22:26:14,854  INFO [Server Action Executor Worker 413349] KerberosServerAction:430 - Processing identities...
2019-06-11 22:31:17,327  WARN [Server Action Executor Worker 413349] ShellCommandUtil:213 - Can not perform chown yarn-ats /etc/security/keytabs/yarn-ats.hbase-master.service.keytab
java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
	at org.apache.ambari.server.utils.ShellCommandUtil.runCommand(ShellCommandUtil.java:495)
	at org.apache.ambari.server.utils.ShellCommandUtil.setFileOwner(ShellCommandUtil.java:210)
	at org.apache.ambari.server.serveraction.kerberos.FinalizeKerberosServerAction.processIdentity(FinalizeKerberosServerAction.java:110)
	at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(KerberosServerAction.java:458)
	at org.apache.ambari.server.serveraction.kerberos.FinalizeKerberosServerAction.execute(FinalizeKerberosServerAction.java:180)
	at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:550)
	at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:466)
	at java.lang.Thread.run(Thread.java:745)
2019-06-11 22:31:17,328 ERROR [Server Action Executor Worker 413349] FinalizeKerberosServerAction:119 - Failed to update the owner of the keytab file at /etc/security/keytabs/yarn-ats.hbase-master.service.keytab to yarn-ats: Cannot perform operation: null
2019-06-11 22:31:17,339  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/yarn-ats.hbase-master.service.keytab to hadoop
2019-06-11 22:31:17,351  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/yarn-ats.hbase-master.service.keytab to owner:'r' and group:''
2019-06-11 22:31:17,392  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/yarn-ats.hbase-client.headless.keytab to yarn-ats
2019-06-11 22:31:17,402  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/yarn-ats.hbase-client.headless.keytab to hadoop
2019-06-11 22:31:17,411  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/yarn-ats.hbase-client.headless.keytab to owner:'r' and group:''
2019-06-11 22:31:17,616  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/smokeuser.headless.keytab to ambari-qa
2019-06-11 22:31:17,627  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/smokeuser.headless.keytab to hadoop
2019-06-11 22:31:17,639  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/smokeuser.headless.keytab to owner:'r' and group:'r'
2019-06-11 22:31:17,681  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/activity-explorer.headless.keytab to null
2019-06-11 22:31:17,682  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/activity-explorer.headless.keytab to null
2019-06-11 22:31:17,694  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/activity-explorer.headless.keytab to owner:'r' and group:'null'
2019-06-11 22:31:17,779  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/spnego.service.keytab to root
2019-06-11 22:31:17,791  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/spnego.service.keytab to hadoop
2019-06-11 22:31:17,803  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/spnego.service.keytab to owner:'r' and group:'r'
2019-06-11 22:31:17,861  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/spark2.headless.keytab to spark
2019-06-11 22:31:17,874  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/spark2.headless.keytab to hadoop
2019-06-11 22:31:17,885  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/spark2.headless.keytab to owner:'r' and group:''
2019-06-11 22:31:17,952  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/hdfs.headless.keytab to hdfs
2019-06-11 22:31:17,961  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/hdfs.headless.keytab to hadoop
2019-06-11 22:31:17,970  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/hdfs.headless.keytab to owner:'r' and group:''
2019-06-11 22:31:18,024  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/spark.service.keytab to spark
2019-06-11 22:31:18,036  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/spark.service.keytab to hadoop
2019-06-11 22:31:18,045  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/spark.service.keytab to owner:'r' and group:''
2019-06-11 22:31:18,188  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/logfeeder.service.keytab to root
2019-06-11 22:31:18,198  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/logfeeder.service.keytab to hadoop
2019-06-11 22:31:18,207  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/logfeeder.service.keytab to owner:'r' and group:''
2019-06-11 22:31:18,282  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/yarn-ats.hbase-regionserver.service.keytab to yarn-ats
2019-06-11 22:31:18,293  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/yarn-ats.hbase-regionserver.service.keytab to hadoop
2019-06-11 22:31:18,303  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/yarn-ats.hbase-regionserver.service.keytab to owner:'r' and group:''
2019-06-11 22:31:18,376  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/hbase.headless.keytab to hbase
2019-06-11 22:31:18,386  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/hbase.headless.keytab to hadoop
2019-06-11 22:31:18,396  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/hbase.headless.keytab to owner:'r' and group:'r'
2019-06-11 22:31:18,442  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/ams-monitor.keytab to ams
2019-06-11 22:31:18,453  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/ams-monitor.keytab to hadoop
2019-06-11 22:31:18,460  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/ams-monitor.keytab to owner:'r' and group:''

By default it has 300seconds at

https://github.com/apache/ambari/blob/release-2.7.3-rc0/ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java#L3730

this needs to be configurable.

 

Issue Links

• links to: GitHub PR#3404
• links to: GitHub Pull Request #44
• links to: GitHub Pull Request #3041