[AMBARI-19593] Hbase, Zookeeper start failed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.5.0
• Assignee: Andrew Onischuk
• Reporter: Andrew Khalymon
• Created: 17/Jan/17 16:37
• Updated: 21/Apr/17 17:56
• Resolved: 19/Jan/17 08:52

Description

Hbase master and zookeeper server start failed with

"href" : "http://172.22.119.57:8080/api/v1/clusters/cl1/requests/19/tasks/354",
"Tasks" : {
"attempt_cnt" : 1,
"cluster_name" : "cl1",
"command" : "START",
"command_detail" : "HBASE_MASTER START",
"end_time" : 1484027047948,
"error_log" : "/var/lib/ambari-agent/data/errors-354.txt",
"exit_code" : 1,
"host_name" : "nat-d7-hxks-ambari-kerb-ad-2-2.openstacklocal",
"id" : 354,
"output_log" : "/var/lib/ambari-agent/data/output-354.txt",
"request_id" : 19,
"role" : "HBASE_MASTER",
"stage_id" : 1,
"start_time" : 1484027045724,
"status" : "FAILED",
"stderr" : "Traceback (most recent call last):\n File \"/var/lib/ambari-agent/cache/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_master.py\", line 157, in <module>\n HbaseMaster().execute()\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 287, in execute\n method(env)\n File \"/var/lib/ambari-agent/cache/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_master.py\", line 84, in start\n self.configure(env) # for security\n File \"/var/lib/ambari-agent/cache/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_master.py\", line 39, in configure\n hbase(name='master')\n File \"/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py\", line 89, in thunk\n return fn(*args, **kwargs)\n File \"/var/lib/ambari-agent/cache/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py\", line 199, in hbase\n owner=params.hbase_user\n File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 155, in _init_\n self.env.run()\n File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 160, in run\n self.run_action(resource, action)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 124, in run_action\n provider_action()\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py\", line 459, in action_create_on_execute\n self.action_delayed(\"create\")\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py\", line 456, in action_delayed\n self.get_hdfs_resource_executor().action_delayed(action_name, self)\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py\", line 247, in action_delayed\n self._assert_valid()\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py\", line 231, in _assert_valid\n self.target_status = self._get_file_status(target)\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py\", line 292, in _get_file_status\n list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py\", line 179, in run_command\n _, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False)\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py\", line 61, in get_user_call_output\n raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])\nresource_management.core.exceptions.ExecutionFailed: Execution of 'curl -sS -L -w '%

{http_code}

' -X GET --negotiate -u : 'http://nat-d7-hxks-ambari-kerb-ad-2-5.openstacklocal:20070/webhdfs/v1/apps/hbase/data?op=GETFILESTATUS&user.name=cstm-hdfs' 1>/tmp/tmpLtFPf0 2>/tmp/tmppl_1zh' returned 7. curl: (7) couldn't connect to host\n000",

"href" : "http://172.22.119.57:8080/api/v1/clusters/cl1/requests/20/tasks/381",
"Tasks" : {
"attempt_cnt" : 1,
"cluster_name" : "cl1",
"command" : "START",
"command_detail" : "ZOOKEEPER_SERVER START",
"end_time" : 1484027166947,
"error_log" : "/var/lib/ambari-agent/data/errors-381.txt",
"exit_code" : 1,
"host_name" : "nat-d7-hxks-ambari-kerb-ad-2-4.openstacklocal",
"id" : 381,
"output_log" : "/var/lib/ambari-agent/data/output-381.txt",
"request_id" : 20,
"role" : "ZOOKEEPER_SERVER",
"stage_id" : 1,
"start_time" : 1484027164691,
"status" : "FAILED",
"stderr" : "2017-01-10 05:46:05,892 - Could not determine stack version for component zookeeper-server by calling '/usr/bin/hdp-select status zookeeper-server > /tmp/tmpVf4jWA'. Return Code: 1, Output: .\nTraceback (most recent call last):\n File \"/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper_server.py\", line 189, in <module>\n ZookeeperServer().execute()\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 287, in execute\n method(env)\n File \"/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper_server.py\", line 57, in start\n zookeeper_service(action='start', upgrade_type=upgrade_type)\n File \"/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py\", line 89, in thunk\n return fn(*args, **kwargs)\n File \"/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper_service.py\", line 41, in zookeeper_service\n user=params.zk_user\n File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 155, in _init_\n self.env.run()\n File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 160, in run\n self.run_action(resource, action)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 124, in run_action\n provider_action()\n File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 262, in action_run\n tries=self.resource.tries, try_sleep=self.resource.try_sleep)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 72, in inner\n result = function(command, **kwargs)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 102, in checked_call\n tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 150, in _call_wrapper\n result = _call(command, **kwargs_copy)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 303, in _call\n raise ExecutionFailed(err_msg, code, out, err)\nresource_management.core.exceptions.ExecutionFailed: Execution of 'source /usr/hdp/current/zookeeper-server/conf/zookeeper-env.sh ; env ZOOCFGDIR=/usr/hdp/current/zookeeper-server/conf ZOOCFG=zoo.cfg /usr/hdp/current/zookeeper-server/bin/zkServer.sh start' returned 126. ######## Hortonworks #############\nThis is MOTD message, added for testing in qe infra\n-su: /usr/hdp/current/zookeeper-server/conf/zookeeper-env.sh: Permission denied\nenv: /usr/hdp/current/zookeeper-server/bin/zkServer.sh: Permission denied",

artifacts: <http://qelog.hortonworks.com/log/nat-d7-hxks-ambari-kerb-ad-2
/test-logs/ambari-kerb-ad/artifacts/screenshots/com.hw.ambari.ui.tests.monitor
ing.admin_page.TestAllSmokeTests/runSmokeTestAfterInstallHadoop/_10_5_56_43_Ch
ecking_smoke_test_for__MAPREDUCE2_service_failed/>

Issue Links

• links to: ReviewBoard