[AMBARI-5066] Ambari Upgrade Test, restart HBase, HBase Client Install failed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Assignee: Andrew Onischuk
• Reporter: Andrew Onischuk
• Created: 13/Mar/14 12:41
• Updated: 13/Mar/14 12:49
• Resolved: 13/Mar/14 12:49

Description

Upgraded from Ambari-1.4.1 Stack 2.0
When running QE upgrade script, which simply installs old version of Ambari, deploys HDP then upgrades Ambari server and runs smoke tests, it failed at restart HBase service.

The script uses API call to stop services:

curl -H "X-Requested-By: ambari" -X PUT -u admin:admin http:/192.168.64.101:8080/api/v1/clusters/cl1/services?ServiceInfo/state=STARTED  -d '{"ServiceInfo": {"state" : "INSTALLED"}}'

Output:

{
  "href" : "http://192.168.64.101:8080/api/v1/clusters/cl1/requests/10",
  "Requests" : {
    "id" : 10,
    "status" : "InProgress"
  }
}

HBase region server failed to stop

stderr: 
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HBASE/package/scripts/hbase_regionserver.py", line 66, in <module>
    HbaseRegionServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 95, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HBASE/package/scripts/hbase_regionserver.py", line 48, in stop
    import params
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HBASE/package/scripts/params.py", line 56, in <module>
    local_dir = substitute_vars(config['configurations']['hbase-site']['hbase.local.dir'], config['configurations']['hbase-site'])
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/substitute_vars.py", line 36, in substitute_vars
    match = pattern.search(result)
TypeError: expected string or buffer
 stdout:
None

/var/lib/ambari-agent/data/errors-71.txt is attached.

The when trying to restart HBase, HBase client failed to install:

stderr: 
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HBASE/package/scripts/hbase_client.py", line 43, in <module>
    HbaseClient().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 95, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HBASE/package/scripts/hbase_client.py", line 30, in install
    self.configure(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HBASE/package/scripts/hbase_client.py", line 33, in configure
    import params
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HBASE/package/scripts/params.py", line 56, in <module>
    local_dir = substitute_vars(config['configurations']['hbase-site']['hbase.local.dir'], config['configurations']['hbase-site'])
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/substitute_vars.py", line 36, in substitute_vars
    match = pattern.search(result)
TypeError: expected string or buffer
 stdout:
2014-03-12 00:03:40,773 - Group['hadoop'] {}
2014-03-12 00:03:40,775 - Modifying group hadoop
2014-03-12 00:03:40,804 - Group['users'] {}
2014-03-12 00:03:40,805 - Modifying group users
2014-03-12 00:03:40,825 - Group['users'] {}
2014-03-12 00:03:40,826 - Modifying group users
2014-03-12 00:03:40,849 - User['ambari-qa'] {'gid': 'hadoop', 'groups': [u'users']}
2014-03-12 00:03:40,849 - Modifying user ambari-qa
2014-03-12 00:03:40,861 - User['tez'] {'gid': 'hadoop', 'groups': [u'users']}
2014-03-12 00:03:40,862 - Adding user User['tez']
2014-03-12 00:03:40,892 - File['/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2014-03-12 00:03:40,899 - Changing permission for /tmp/changeUid.sh from 755 to 555
2014-03-12 00:03:40,899 - Execute['/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'}
2014-03-12 00:03:40,907 - Skipping Execute['/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] due to not_if
2014-03-12 00:03:40,907 - User['hbase'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2014-03-12 00:03:40,908 - Modifying user hbase
2014-03-12 00:03:40,915 - File['/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2014-03-12 00:03:40,916 - Execute['/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/hadoop/hbase 2>/dev/null'] {'not_if': 'test $(id -u hbase) -gt 1000'}
2014-03-12 00:03:40,923 - Skipping Execute['/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/hadoop/hbase 2>/dev/null'] due to not_if
2014-03-12 00:03:40,924 - Group['nagios'] {}
2014-03-12 00:03:40,924 - Modifying group nagios
2014-03-12 00:03:40,948 - User['nagios'] {'gid': 'nagios'}
2014-03-12 00:03:40,948 - Modifying user nagios
2014-03-12 00:03:40,958 - User['hcat'] {'gid': 'hadoop'}
2014-03-12 00:03:40,958 - Modifying user hcat
2014-03-12 00:03:40,965 - User['hcat'] {'gid': 'hadoop'}
2014-03-12 00:03:40,965 - Modifying user hcat
2014-03-12 00:03:40,971 - User['hive'] {'gid': 'hadoop'}
2014-03-12 00:03:40,971 - Modifying user hive
2014-03-12 00:03:40,978 - User['yarn'] {'gid': 'hadoop'}
2014-03-12 00:03:40,978 - Modifying user yarn
2014-03-12 00:03:40,986 - Group['nobody'] {}
2014-03-12 00:03:40,986 - Modifying group nobody
2014-03-12 00:03:41,004 - Group['nobody'] {}
2014-03-12 00:03:41,005 - Modifying group nobody
2014-03-12 00:03:41,025 - User['nobody'] {'gid': 'hadoop', 'groups': [u'nobody']}
2014-03-12 00:03:41,025 - Modifying user nobody
2014-03-12 00:03:41,047 - User['nobody'] {'gid': 'hadoop', 'groups': [u'nobody']}
2014-03-12 00:03:41,048 - Modifying user nobody
2014-03-12 00:03:41,056 - User['hdfs'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2014-03-12 00:03:41,056 - Modifying user hdfs
2014-03-12 00:03:41,065 - User['mapred'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2014-03-12 00:03:41,066 - Modifying user mapred
2014-03-12 00:03:41,074 - User['zookeeper'] {'gid': 'hadoop'}
2014-03-12 00:03:41,075 - Modifying user zookeeper
2014-03-12 00:03:41,084 - Package['unzip'] {}
2014-03-12 00:03:41,126 - Installing package unzip ('/usr/bin/yum -d 0 -e 0 -y install unzip')
2014-03-12 00:03:43,299 - Package['net-snmp'] {}
2014-03-12 00:03:43,316 - Skipping installing existent package net-snmp
2014-03-12 00:03:43,317 - Package['net-snmp-utils'] {}
2014-03-12 00:03:43,333 - Skipping installing existent package net-snmp-utils
2014-03-12 00:03:43,425 - Repository['HDP-2.0.6'] {'action': ['create'], 'mirror_list': None, 'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.0.6.0', 'repo_file_name': 'HDP'}
2014-03-12 00:03:43,431 - File['/etc/yum.repos.d/HDP.repo'] {'content': InlineTemplate(...)}
2014-03-12 00:03:43,433 - Writing File['/etc/yum.repos.d/HDP.repo'] because contents don't match
2014-03-12 00:03:43,433 - Package['hbase'] {}
2014-03-12 00:03:43,450 - Skipping installing existent package hbase

see /var/lib/ambari-agent/data/errors-85.txt

Issue Links

• links to: ReviewBoard