[AMBARI-18184] Hive Metastore restart failed during EU with 'Internal credentials cache error' while running kinit

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: ambari-server
• Labels: 240RMApproved
• Assignee: Jonathan Hurley
• Reporter: Jonathan Hurley
• Created: 17/Aug/16 14:49
• Updated: 18/Aug/16 21:33
• Resolved: 18/Aug/16 17:03

Description

ambari-server --hash
8250e90dc9ebcf1bd3dbac9b9eca8a6e21e073c9
ambari-server-2.4.0.0-1127.x86_64

Observed this issue in one EU run with below steps:

1. Install HDP-2.4.0.0 cluster with Ambari 2.2.1.1 (secure, HA cluster)
2. Upgrade Ambari to 2.4.0.0
3. Perform EU to 2.4.2.0 and let it complete
4. Start EU to 2.5.0.0

Observed below error during Hive Metastore restart

Traceback (most recent call last):
  File \"/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py\", line 254, in <module>
    HiveMetastore().execute()
  File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 280, in execute
    method(env)
  File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 696, in restart
    self.pre_upgrade_restart(env, upgrade_type=upgrade_type)
  File \"/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py\", line 114, in pre_upgrade_restart
    self.upgrade_schema(env)
  File \"/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py\", line 193, in upgrade_schema
    Execute(kinit_command,user=params.smokeuser)
  File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 155, in __init__
    self.env.run()
  File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 160, in run
    self.run_action(resource, action)
  File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 124, in run_action
    provider_action()
  File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 273, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 71, in inner
    result = function(command, **kwargs)
  File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 93, in checked_call
    tries=tries, try_sleep=try_sleep)
  File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 141, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 294, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab [e-mail]; ' returned 1. kinit: Internal credentials cache error while storing credentials while getting initial credentials"

A retry of the above failed task was successful and then EU proceeded to completion

Issue Links

• links to: Reviewboard