Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-10118

Enabling ResourceManager HA failed in Service Check for HDFS

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      • Deployed Ubuntu cluster on EC2 with 7 hosts
      • Installed HDP 2.2.3.0-2611
      • Added services, HDFS, MR, Yarn, Pig, Tez, Hive, HBASE, ZK, Oozie, Knox, AMS
      • Enabled Resource Manager HA
      • HA Wizard does not complete because HDFS service check fails

      Cluster is still live, c2-54-152-254-11.compute-1.amazonaws.com:8080

      Fail: Execution of 'hadoop --config /etc/hadoop/conf fs -mkdir /tmp' returned 1. mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x

      2015-03-17 23:07:50,695 - u"ExecuteHadoop['dfsadmin -safemode get | grep OFF']"

      {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'conf_dir': '/etc/hadoop/conf', 'logoutput': True, 'try_sleep': 3, 'tries': 20, 'user': 'ambari-qa'}

      2015-03-17 23:07:50,698 - u"Execute['hadoop --config /etc/hadoop/conf dfsadmin -safemode get | grep OFF']" {'logoutput': True, 'try_sleep': 3, 'environment': {}, 'tries': 20, 'user': 'ambari-qa', 'path': ['/usr/hdp/current/hadoop-client/bin']}
      DEPRECATED: Use of this script to execute hdfs command is deprecated.
      Instead use the hdfs command for it.

      Safe mode is OFF in ip-172-31-15-115.ec2.internal/172.31.15.115:8020
      2015-03-17 23:07:55,117 - u"ExecuteHadoop['fs -mkdir /tmp']"

      {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'conf_dir': '/etc/hadoop/conf', 'logoutput': True, 'not_if': "ambari-sudo.sh su ambari-qa -l -s /bin/bash -c '/usr/hdp/current/hadoop-client/bin/hadoop --config /etc/hadoop/conf fs -test -e /tmp'", 'try_sleep': 3, 'tries': 5, 'user': 'ambari-qa'}

      2015-03-17 23:07:59,346 - u"Execute['hadoop --config /etc/hadoop/conf fs -mkdir /tmp']" {'logoutput': True, 'try_sleep': 3, 'environment': {}, 'tries': 5, 'user': 'ambari-qa', 'path': ['/usr/hdp/current/hadoop-client/bin']}
      mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
      2015-03-17 23:08:03,570 - Retrying after 3 seconds. Reason: Execution of 'hadoop --config /etc/hadoop/conf fs -mkdir /tmp' returned 1. mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
      mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
      2015-03-17 23:08:10,923 - Retrying after 3 seconds. Reason: Execution of 'hadoop --config /etc/hadoop/conf fs -mkdir /tmp' returned 1. mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
      mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
      2015-03-17 23:08:18,163 - Retrying after 3 seconds. Reason: Execution of 'hadoop --config /etc/hadoop/conf fs -mkdir /tmp' returned 1. mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
      mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
      2015-03-17 23:08:25,417 - Retrying after 3 seconds. Reason: Execution of 'hadoop --config /etc/hadoop/conf fs -mkdir /tmp' returned 1. mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
      mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
      2015-03-17 23:08:32,916 - Error while executing command 'service_check':
      Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 214, in execute
      method(env)
      File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/service_check.py", line 62, in service_check
      bin_dir=params.hadoop_bin_dir
      File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in _init_
      self.env.run()
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
      self.run_action(resource, action)
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
      provider_action()
      File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/execute_hadoop.py", line 55, in action_run
      environment = self.resource.environment,
      File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in _init_
      self.env.run()
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
      self.run_action(resource, action)
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
      provider_action()
      File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 274, in action_run
      raise ex
      Fail: Execution of 'hadoop --config /etc/hadoop/conf fs -mkdir /tmp' returned 1. mkdir: Permission denied: user=ambari-qa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                aonishuk Andrew Onischuk
                Reporter:
                aonishuk Andrew Onischuk
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: