Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
2.2.0
-
None
Description
The basic problem appears to be a concurrency problem, in which keytabs are sometimes generated incorrectly. This results in certain keytabs across the cluster having different created/modified times in the keytabs, which can cause Kerberos Preauthentication failures, such as the following:
resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa-clusterone@EXAMPLE.COM;' returned 1. kinit: Password incorrect while getting initial credentials Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py", line 179, in <module> ZookeeperServer().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 217, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py", line 55, in start zookeeper_service(action='start', upgrade_type=upgrade_type) File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py", line 54, in zookeeper_service user=params.smokeuser File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg)
The basic underlying problem appears to be invalid keytabs. Looking to the keytabs on the failing node:
[root@rwntest-2 keytabs]# klist -kt smokeuser.headless.keytab Keytab name: FILE:smokeuser.headless.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM 1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM 1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM 1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM 1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM
whereas other nodes in the cluster report the following value for the same keytab:
[root@rwntest-4 keytabs]# klist -kt smokeuser.headless.keytab Keytab name: FILE:smokeuser.headless.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM 4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM 4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM 4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM 4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM
It looks like these keytabs should have the same timestamps, and this is causing the Kerberos failures at cluster start up.