[AMBARI-20052] Spark2 service check is failing in secure cluster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Eugene Chekanskiy
• Reporter: Yesha Vora
• Created: 16/Feb/17 19:11
• Updated: 20/Feb/17 15:39
• Resolved: 20/Feb/17 14:57

Description

Spark2 Service check is failing in secure cluster.

errors-411.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/service_check.py", line 62, in <module>
    SparkServiceCheck().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 313, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/service_check.py", line 36, in service_check
    livy_kinit_cmd = format("{kinit_path_local} -kt {smoke_user_keytab} {smokeuser_principal}; ")
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py", line 95, in format
    return ConfigurationFormatter().format(format_string, args, **result)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py", line 59, in format
    result_protected = self.vformat(format_string, args, all_params)
  File "/usr/lib64/python2.7/string.py", line 549, in vformat
    result = self._vformat(format_string, args, kwargs, used_args, 2)
  File "/usr/lib64/python2.7/string.py", line 571, in _vformat
    obj, arg_used = self.get_field(field_name, args, kwargs)
  File "/usr/lib64/python2.7/string.py", line 632, in get_field
    obj = self.get_value(first, args, kwargs)
  File "/usr/lib64/python2.7/string.py", line 591, in get_value
    return kwargs[key]
  File "/usr/lib/python2.6/site-packages/resource_management/core/utils.py", line 57, in __getitem__
    return self._convert_value(self._dict[name])
KeyError: 'smoke_user_keytab'

output-411.txt

2017-02-16 18:52:44,887 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-02-16 18:52:44,904 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/spark2.headless.keytab [e-mail]; '] {'user': 'spark'}

Command failed after 1 tries

As per above logs, kinit operation failed on a host. I was able to run same kinit command successfully from same host manually.

[root@xxx ~]$ sudo su spark
bash-4.2$ /usr/bin/kinit -kt /etc/security/keytabs/spark2.headless.keytab [e-mail]
bash-4.2$ echo $?
0

Issue Links

• links to: review