[AMBARI-15165] HDFS Datanode won't start in secure cluster - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Not A Problem
Affects Version/s: 2.2.0
Fix Version/s: None
Component/s: ambari-web
Labels:
None
Environment:
Hide

$ cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) $ uname -a Linux dev09-ost-hivetest-h-hb02.td.local 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Show
$ cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) $ uname -a Linux dev09-ost-hivetest-h-hb02.td.local 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Description

This issue sounds related, but I'm on the newer version which should include this patch already: https://issues.apache.org/jira/browse/AMBARI-12355

When I attempt to Kerberoize a HDP cluster the startup of the HDFS datanode fails quietly. Nothing telling in the logs, see the referenced below ambari-agent errors log.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 167, in <module>
    DataNode().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 62, in start
    datanode(action="start")
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py", line 72, in datanode
    create_log_dir=True
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py", line 267, in service
    Execute(daemon_cmd, not_if=process_id_exists_command, environment=hadoop_env_exports)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh  -H -E /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode' returned 1. starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-dev09-ost-hivetest-h-hb02.td.local.out
stdout:   /var/lib/ambari-agent/data/output-228.txt

2016-02-24 10:51:14,841 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-02-24 10:51:14,841 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-02-24 10:51:14,841 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-02-24 10:51:14,877 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-02-24 10:51:14,878 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-02-24 10:51:14,910 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-02-24 10:51:14,910 - Ensuring that hadoop has the correct symlink structure
2016-02-24 10:51:14,910 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-02-24 10:51:15,091 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-02-24 10:51:15,091 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-02-24 10:51:15,091 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-02-24 10:51:15,120 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-02-24 10:51:15,121 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-02-24 10:51:15,162 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-02-24 10:51:15,162 - Ensuring that hadoop has the correct symlink structure
2016-02-24 10:51:15,162 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-02-24 10:51:15,164 - Group['hadoop'] {}
2016-02-24 10:51:15,165 - Group['users'] {}
2016-02-24 10:51:15,166 - Group['knox'] {}
2016-02-24 10:51:15,166 - User['hive'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2016-02-24 10:51:15,167 - User['zookeeper'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2016-02-24 10:51:15,168 - User['ams'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2016-02-24 10:51:15,168 - User['ambari-qa'] {'gid': 'hadoop', 'groups': [u'users']}
2016-02-24 10:51:15,169 - User['tez'] {'gid': 'hadoop', 'groups': [u'users']}
2016-02-24 10:51:15,170 - User['hdfs'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2016-02-24 10:51:15,171 - User['yarn'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2016-02-24 10:51:15,172 - User['hcat'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2016-02-24 10:51:15,172 - User['mapred'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2016-02-24 10:51:15,173 - User['hbase'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2016-02-24 10:51:15,174 - User['knox'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2016-02-24 10:51:15,175 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-02-24 10:51:15,177 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2016-02-24 10:51:15,182 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2016-02-24 10:51:15,183 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'}
2016-02-24 10:51:15,184 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-02-24 10:51:15,185 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2016-02-24 10:51:15,190 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2016-02-24 10:51:15,191 - Group['hdfs'] {'ignore_failures': False}
2016-02-24 10:51:15,191 - User['hdfs'] {'ignore_failures': False, 'groups': [u'hadoop', u'hdfs']}
2016-02-24 10:51:15,192 - Directory['/etc/hadoop'] {'mode': 0755}
2016-02-24 10:51:15,210 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'}
2016-02-24 10:51:15,211 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777}
2016-02-24 10:51:15,224 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2016-02-24 10:51:15,237 - Skipping Execute[('setenforce', '0')] due to not_if
2016-02-24 10:51:15,237 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-02-24 10:51:15,240 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}
2016-02-24 10:51:15,240 - Changing owner for /var/run/hadoop from 1006 to root
2016-02-24 10:51:15,240 - Changing group for /var/run/hadoop from 1001 to root
2016-02-24 10:51:15,240 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}
2016-02-24 10:51:15,245 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'}
2016-02-24 10:51:15,247 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'root'}
2016-02-24 10:51:15,248 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2016-02-24 10:51:15,259 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
2016-02-24 10:51:15,260 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2016-02-24 10:51:15,261 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2016-02-24 10:51:15,266 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2016-02-24 10:51:15,271 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2016-02-24 10:51:15,467 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-02-24 10:51:15,468 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-02-24 10:51:15,468 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-02-24 10:51:15,501 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-02-24 10:51:15,501 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-02-24 10:51:15,534 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-02-24 10:51:15,534 - Ensuring that hadoop has the correct symlink structure
2016-02-24 10:51:15,534 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-02-24 10:51:15,536 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-02-24 10:51:15,536 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-02-24 10:51:15,537 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-02-24 10:51:15,565 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-02-24 10:51:15,566 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-02-24 10:51:15,595 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-02-24 10:51:15,596 - Ensuring that hadoop has the correct symlink structure
2016-02-24 10:51:15,596 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-02-24 10:51:15,605 - Directory['/etc/security/limits.d'] {'owner': 'root', 'group': 'root', 'recursive': True}
2016-02-24 10:51:15,612 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2016-02-24 10:51:15,613 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-02-24 10:51:15,626 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml
2016-02-24 10:51:15,627 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-02-24 10:51:15,638 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-02-24 10:51:15,649 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml
2016-02-24 10:51:15,650 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-02-24 10:51:15,657 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-02-24 10:51:15,658 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2016-02-24 10:51:15,669 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
2016-02-24 10:51:15,669 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-02-24 10:51:15,677 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-02-24 10:51:15,688 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml
2016-02-24 10:51:15,689 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-02-24 10:51:15,697 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-02-24 10:51:15,708 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
2016-02-24 10:51:15,709 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-02-24 10:51:15,770 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2016-02-24 10:51:15,781 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
2016-02-24 10:51:15,782 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2016-02-24 10:51:15,810 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'root'}
2016-02-24 10:51:15,811 - Directory['/var/lib/hadoop-hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0751, 'recursive': True}
2016-02-24 10:51:15,817 - Host contains mounts: ['/sys', '/proc', '/dev', '/sys/kernel/security', '/dev/shm', '/dev/pts', '/run', '/sys/fs/cgroup', '/sys/fs/cgroup/systemd', '/sys/fs/pstore', '/sys/fs/cgroup/perf_event', '/sys/fs/cgroup/memory', '/sys/fs/cgroup/devices', '/sys/fs/cgroup/cpuset', '/sys/fs/cgroup/hugetlb', '/sys/fs/cgroup/freezer', '/sys/fs/cgroup/blkio', '/sys/fs/cgroup/cpu,cpuacct', '/sys/fs/cgroup/net_cls', '/sys/kernel/config', '/', '/proc/sys/fs/binfmt_misc', '/dev/mqueue', '/sys/kernel/debug', '/dev/hugepages', '/run/user/0', '/run/user/1000', '/proc/sys/fs/binfmt_misc'].
2016-02-24 10:51:15,817 - Mount point for directory /hadoop/hdfs/data is /
2016-02-24 10:51:15,817 - File['/var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist'] {'content': '\n# This file keeps track of the last known mount-point for each DFS data dir.\n# It is safe to delete, since it will get regenerated the next time that the DataNode starts.\n# However, it is not advised to delete this file since Ambari may\n# re-create a DFS data dir that used to be mounted on a drive but is now mounted on the root.\n# Comments begin with a hash (#) symbol\n# data_dir,mount_point\n/hadoop/hdfs/data,/\n', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2016-02-24 10:51:15,819 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755}
2016-02-24 10:51:15,819 - Changing owner for /var/run/hadoop from 0 to hdfs
2016-02-24 10:51:15,819 - Changing group for /var/run/hadoop from 0 to hadoop
2016-02-24 10:51:15,819 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'recursive': True}
2016-02-24 10:51:15,820 - Directory['/var/log/hadoop/hdfs'] {'owner': 'hdfs', 'recursive': True}
2016-02-24 10:51:15,820 - File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'}
2016-02-24 10:51:15,833 - Deleting File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid']
2016-02-24 10:51:15,833 - Execute['ambari-sudo.sh  -H -E /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode'] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'}

When I attempted to run the hdfs ... datanode command directly like so:

strace -s 2000 -o ~/slog.txt /usr/hdp/2.3.4.0-3485/hadoop-hdfs/bin/hdfs --config /usr/hdp/current/hadoop-client/conf datanode

I noticed this section which mentions to additional log files I hadn't see before.

read(255, "#!/usr/bin/env bash\n\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# Environment Variables\n#\n#   JSVC_HOME  home directory of jsvc binary.  Required for starting secure\n#              datanode.\n#\n#   JSVC_OUTFILE  path to jsvc output file.  Defaults to\n#                 $HADOOP_LOG_DIR/jsvc.out.\n#\n#   JSVC_ERRFILE  path to jsvc error file.  Defaults to $HADOOP_LOG_DIR/jsvc.err.\n\nbin=`which $0`\nbin=`dirname ${bin}`\nbin=`cd \"$bin\" > /dev/null; pwd`\n\nDEFAULT_LIBEXEC_DIR=\"$bin\"/../libexec\n\nif [ -n \"$HADOOP_HOME\" ]; then\n  DEFAULT_LIBEXEC_DIR=\"$HADOOP_HOME\"/libexec\nfi\n\nHADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}\n. $HADOOP_LIBEXEC_DIR/hdfs-config.sh\n\nfunction print_usage(){\n  echo \"Usage: hdfs [--config confdir] [--loglevel loglevel] COMMAND\"\n  echo \"       where COMMAND is one of:\"\n  echo \"  dfs                  run a filesystem command on the file systems supported in Hadoop.\"\n  echo \"  classpath            prints the classpath\"\n  echo \"  namenode -format     format the DFS filesystem\"\n  echo \"  secondarynamenode    run the DFS secondary namenode\"\n  echo \"  namenode             run the DFS namenode\"\n  echo \"  journalnode          run the DFS journalnode\"\n  echo \"  zkfc                 run the ZK Failover Controller daemon\"\n  echo"..., 8192) = 8192

Specifically these files:

/var/log/hadoop/hdfs/jsvc.out
/var/log/hadoop/hdfs/jsvc.err

In looking in the jsvc.err file I found this:

STARTUP_MSG:   build = git@github.com:hortonworks/hadoop.git -r ef0582ca14b8177a3cbb6376807545272677d730; compiled by 'jenkins' on 2015-12-16T03:01Z
STARTUP_MSG:   java = 1.8.0_60
************************************************************/
16/02/24 11:30:18 INFO datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
16/02/24 11:30:18 FATAL datanode.DataNode: Exception in secureMain
java.io.IOException: Login failure for dn/host-192-168-114-49.td.local@<REDACTED KERBEROS REALM> from keytab /etc/security/keytabs/dn.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:275)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2296)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2345)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2526)
        at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.start(SecureDataNodeStarter.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:953)
        ... 10 more
16/02/24 11:30:18 INFO util.ExitUtil: Exiting with status 1
16/02/24 11:30:18 INFO datanode.DataNode: SHUTDOWN_MSG:
/************************************************************

Attachments

Issue Links

is related to

AMBARI-12355 DATANODE START failed on secure cluster

Resolved

Activity

People

Assignee:: Unassigned

Reporter:: Sam Mingolelli

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 24/Feb/16 16:58

Updated:: 24/Feb/16 23:35

Resolved:: 24/Feb/16 23:35