[AMBARI-22492] A bad WebHDFS request is issued when starting Hive Metastore

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.6.1
• Assignee: Attila Magyar
• Reporter: David Tucker
• Created: 21/Nov/17 20:21
• Updated: 07/Dec/17 12:28
• Resolved: 06/Dec/17 21:19

Description

When deploying a cluster with Hive (1.2.1000, in this case), Ambari issues an incorrect WebHDFS request:

2017-11-20 20:21:21,839 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://dtucker-rjy277d.west.isilon.com:8082/webhdfs/v1?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpzipnFe 2>/tmp/tmpbGajx_''] {'logoutput': None, 'quiet': False}

That command should be:

curl -sS -L -w '%{http_code}' -X GET 'http://dtucker-rjy277d.west.isilon.com:8082/webhdfs/v1/?op=GETFILESTATUS&user.name=hdfs'

(Note the / after v1.)

On Apache, this results in a FileNotFoundException (404); however, this causes problems with OneFS which returns an IllegalArgumentException (400) and prevents the Metastore from starting.

Here is the full traceback:

Traceback (most recent call last):
 File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 203, in <module>
   HiveMetastore().execute()
 File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
   method(env)
 File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 54, in start
   self.configure(env)
 File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 120, in locking_configure
   original_configure(obj, *args, **kw)
 File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 72, in configure
   hive(name = 'metastore')
 File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
   return fn(*args, **kwargs)
 File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 343, in hive
   mode = 01777)
 File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
   self.env.run()
 File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
   self.run_action(resource, action)
 File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
   provider_action()
 File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 604, in action_create_on_execute
   self.action_delayed("create")
 File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 601, in action_delayed
   self.get_hdfs_resource_executor().action_delayed(action_name, self)
 File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 328, in action_delayed
   self._assert_valid()
 File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 287, in _assert_valid
   self.target_status = self._get_file_status(target)
 File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 430, in _get_file_status
   list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
 File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 177, in run_command
   return self._run_command(*args, **kwargs)
 File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 248, in _run_command
   raise WebHDFSCallException(err_msg, result_dict)

Issue Links

• links to: review board