[AMBARI-10118] Enabling ResourceManager HA failed in Service Check for HDFS

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Assignee: Andrew Onischuk
• Reporter: Andrew Onischuk
• Created: 18/Mar/15 12:30
• Updated: 18/Mar/15 14:50
• Resolved: 18/Mar/15 12:40

Description

• Deployed Ubuntu cluster on EC2 with 7 hosts
• Installed HDP 2.2.3.0-2611
• Added services, HDFS, MR, Yarn, Pig, Tez, Hive, HBASE, ZK, Oozie, Knox, AMS
• Enabled Resource Manager HA
• HA Wizard does not complete because HDFS service check fails

Cluster is still live, c2-54-152-254-11.compute-1.amazonaws.com:8080

Fail: Execution of 'hadoop --config /etc/hadoop/conf fs -mkdir /tmp' returned 1. mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x

2015-03-17 23:07:50,695 - u"ExecuteHadoop['dfsadmin -safemode get | grep OFF']"

{'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'conf_dir': '/etc/hadoop/conf', 'logoutput': True, 'try_sleep': 3, 'tries': 20, 'user': 'ambari-qa'}

2015-03-17 23:07:50,698 - u"Execute['hadoop --config /etc/hadoop/conf dfsadmin -safemode get | grep OFF']" {'logoutput': True, 'try_sleep': 3, 'environment': {}, 'tries': 20, 'user': 'ambari-qa', 'path': ['/usr/hdp/current/hadoop-client/bin']}
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Safe mode is OFF in ip-172-31-15-115.ec2.internal/172.31.15.115:8020
2015-03-17 23:07:55,117 - u"ExecuteHadoop['fs -mkdir /tmp']"

{'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'conf_dir': '/etc/hadoop/conf', 'logoutput': True, 'not_if': "ambari-sudo.sh su ambari-qa -l -s /bin/bash -c '/usr/hdp/current/hadoop-client/bin/hadoop --config /etc/hadoop/conf fs -test -e /tmp'", 'try_sleep': 3, 'tries': 5, 'user': 'ambari-qa'}

2015-03-17 23:07:59,346 - u"Execute['hadoop --config /etc/hadoop/conf fs -mkdir /tmp']" {'logoutput': True, 'try_sleep': 3, 'environment': {}, 'tries': 5, 'user': 'ambari-qa', 'path': ['/usr/hdp/current/hadoop-client/bin']}
mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
2015-03-17 23:08:03,570 - Retrying after 3 seconds. Reason: Execution of 'hadoop --config /etc/hadoop/conf fs -mkdir /tmp' returned 1. mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
2015-03-17 23:08:10,923 - Retrying after 3 seconds. Reason: Execution of 'hadoop --config /etc/hadoop/conf fs -mkdir /tmp' returned 1. mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
2015-03-17 23:08:18,163 - Retrying after 3 seconds. Reason: Execution of 'hadoop --config /etc/hadoop/conf fs -mkdir /tmp' returned 1. mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
2015-03-17 23:08:25,417 - Retrying after 3 seconds. Reason: Execution of 'hadoop --config /etc/hadoop/conf fs -mkdir /tmp' returned 1. mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
2015-03-17 23:08:32,916 - Error while executing command 'service_check':
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 214, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/service_check.py", line 62, in service_check
bin_dir=params.hadoop_bin_dir
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in _init_
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/execute_hadoop.py", line 55, in action_run
environment = self.resource.environment,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in _init_
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 274, in action_run
raise ex
Fail: Execution of 'hadoop --config /etc/hadoop/conf fs -mkdir /tmp' returned 1. mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x

Issue Links

• links to: ReviewBoard