Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-25311

FinalizeKerberosServerAction timeout has to be configurable

    XMLWordPrintableJSON

Details

    Description

      During the HDP Upgrade to 3.1 there is re-generate keytab operation which is timing out in some cases. here is the example log.

      2019-06-11 22:26:14,854  INFO [Server Action Executor Worker 413349] KerberosServerAction:430 - Processing identities...
2019-06-11 22:31:17,327  WARN [Server Action Executor Worker 413349] ShellCommandUtil:213 - Can not perform chown yarn-ats /etc/security/keytabs/yarn-ats.hbase-master.service.keytab
java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
	at org.apache.ambari.server.utils.ShellCommandUtil.runCommand(ShellCommandUtil.java:495)
	at org.apache.ambari.server.utils.ShellCommandUtil.setFileOwner(ShellCommandUtil.java:210)
	at org.apache.ambari.server.serveraction.kerberos.FinalizeKerberosServerAction.processIdentity(FinalizeKerberosServerAction.java:110)
	at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(KerberosServerAction.java:458)
	at org.apache.ambari.server.serveraction.kerberos.FinalizeKerberosServerAction.execute(FinalizeKerberosServerAction.java:180)
	at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:550)
	at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:466)
	at java.lang.Thread.run(Thread.java:745)
2019-06-11 22:31:17,328 ERROR [Server Action Executor Worker 413349] FinalizeKerberosServerAction:119 - Failed to update the owner of the keytab file at /etc/security/keytabs/yarn-ats.hbase-master.service.keytab to yarn-ats: Cannot perform operation: null
2019-06-11 22:31:17,339  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/yarn-ats.hbase-master.service.keytab to hadoop
2019-06-11 22:31:17,351  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/yarn-ats.hbase-master.service.keytab to owner:'r' and group:''
2019-06-11 22:31:17,392  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/yarn-ats.hbase-client.headless.keytab to yarn-ats
2019-06-11 22:31:17,402  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/yarn-ats.hbase-client.headless.keytab to hadoop
2019-06-11 22:31:17,411  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/yarn-ats.hbase-client.headless.keytab to owner:'r' and group:''
2019-06-11 22:31:17,616  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/smokeuser.headless.keytab to ambari-qa
2019-06-11 22:31:17,627  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/smokeuser.headless.keytab to hadoop
2019-06-11 22:31:17,639  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/smokeuser.headless.keytab to owner:'r' and group:'r'
2019-06-11 22:31:17,681  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/activity-explorer.headless.keytab to null
2019-06-11 22:31:17,682  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/activity-explorer.headless.keytab to null
2019-06-11 22:31:17,694  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/activity-explorer.headless.keytab to owner:'r' and group:'null'
2019-06-11 22:31:17,779  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/spnego.service.keytab to root
2019-06-11 22:31:17,791  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/spnego.service.keytab to hadoop
2019-06-11 22:31:17,803  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/spnego.service.keytab to owner:'r' and group:'r'
2019-06-11 22:31:17,861  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/spark2.headless.keytab to spark
2019-06-11 22:31:17,874  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/spark2.headless.keytab to hadoop
2019-06-11 22:31:17,885  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/spark2.headless.keytab to owner:'r' and group:''
2019-06-11 22:31:17,952  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/hdfs.headless.keytab to hdfs
2019-06-11 22:31:17,961  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/hdfs.headless.keytab to hadoop
2019-06-11 22:31:17,970  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/hdfs.headless.keytab to owner:'r' and group:''
2019-06-11 22:31:18,024  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/spark.service.keytab to spark
2019-06-11 22:31:18,036  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/spark.service.keytab to hadoop
2019-06-11 22:31:18,045  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/spark.service.keytab to owner:'r' and group:''
2019-06-11 22:31:18,188  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/logfeeder.service.keytab to root
2019-06-11 22:31:18,198  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/logfeeder.service.keytab to hadoop
2019-06-11 22:31:18,207  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/logfeeder.service.keytab to owner:'r' and group:''
2019-06-11 22:31:18,282  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/yarn-ats.hbase-regionserver.service.keytab to yarn-ats
2019-06-11 22:31:18,293  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/yarn-ats.hbase-regionserver.service.keytab to hadoop
2019-06-11 22:31:18,303  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/yarn-ats.hbase-regionserver.service.keytab to owner:'r' and group:''
2019-06-11 22:31:18,376  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/hbase.headless.keytab to hbase
2019-06-11 22:31:18,386  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/hbase.headless.keytab to hadoop
2019-06-11 22:31:18,396  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/hbase.headless.keytab to owner:'r' and group:'r'
2019-06-11 22:31:18,442  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:114 - Updated the owner of the keytab file at /etc/security/keytabs/ams-monitor.keytab to ams
2019-06-11 22:31:18,453  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:128 - Updated the group of the keytab file at /etc/security/keytabs/ams-monitor.keytab to hadoop
2019-06-11 22:31:18,460  INFO [Server Action Executor Worker 413349] FinalizeKerberosServerAction:145 - Updated the access mode of the keytab file at /etc/security/keytabs/ams-monitor.keytab to owner:'r' and group:''

      By default it has 300seconds at

      https://github.com/apache/ambari/blob/release-2.7.3-rc0/ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java#L3730

      this needs to be configurable.

       

      Attachments

        Activity

          People

            apappu@hortonworks.com amarnath reddy pappu
            apappu@hortonworks.com amarnath reddy pappu
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 2h 50m
                2h 50m