[AMBARI-17284] Hive Metastore Upgrade Fails Because Of Missing Hive Interactive Directory

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: ambari-server
• Assignee: Jonathan Hurley
• Reporter: Jonathan Hurley
• Created: 16/Jun/16 20:57
• Updated: 17/Jun/16 05:45
• Resolved: 17/Jun/16 03:34

Description

During an upgrade from HDP 2.3/2.4 to HDP 2.5, the following error is seen:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 254, in <module>
    HiveMetastore().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 257, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 662, in restart
    self.pre_upgrade_restart(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 114, in pre_upgrade_restart
    self.upgrade_schema(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 189, in upgrade_schema
    self.configure(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 73, in configure
    hive(name = 'metastore')
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 274, in hive
    jdbc_connector(params.target_hive2, params.hive2_previous_jdbc_jar)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 463, in jdbc_connector
    sudo = True)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 293, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'cp --remove-destination /var/lib/ambari-agent/tmp/mysql-connector-java.jar /usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar' returned 1. cp: cannot create regular file `/usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar': No such file or directory

This is because with HDP 2.5, Hive introduced a new component called hive-server2-hive2:

/usr/hdp/current/hive-metastore -> /usr/hdp/2.5.0.0-1234/hive
/usr/hdp/current/hive-server2 -> /usr/hdp/2.5.0.0-1234/hive
/usr/hdp/current/hive-server2-hive2 -> /usr/hdp/2.5.0.0-1234/hive2

Normally, this would be fine. However, we must use the schematool from hive2 and not hive:

/usr/hdp/2.5.0.0-1234/hive2/bin/schematool

Since Metastore is uggraded first, it's trying to write the custom JDBC connector to
/usr/hdp/current/hive-server2-hive2/lib

But since this component doesn't exist or hasn't been hdp-select'd yet, it fails.

Issue Links

• links to: Reviewboard