[BIGTOP-1336] Puppet recipes failed to deploy kerberos enabled hadoop cluster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 0.8.0
• Component/s: deployment
• Assignee: Unassigned
• Reporter: Evans Ye
• Created: 16/Jun/14 16:19
• Updated: 23/Jun/14 04:02
• Resolved: 23/Jun/14 04:02

Description

Here are some missing dependency setting in our puppet recipes in order to get kerberos enabled on the hadoop cluster.

The first one is that kerberos principal for hdfs user hasn't been created before formatting namenode, which cause the namenode formatting process failed.

The second one is that /etc/default/hadoop-hdfs-datanode doesn't get ready before starting up datanodes and results in datanodes failed to startup.
The datanode error log:

2014-06-16 15:10:10,711 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.lang.RuntimeException: Cannot start secure cluster without privileged resources.

Here's the reproduce steps using vagrant-puppet:
1.) Enable kerberos on the hadoop cluster.

$ vim bigtop-deploy/vm/vagrant-puppet/provision.sh

Add kerberos definitions.

cat > /bigtop-puppet/config/site.csv << EOF
hadoop_head_node,$1
hadoop_storage_dirs,/data/1,/data/2
bigtop_yumrepo_uri,http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64
jdk_package_name,java-1.7.0-openjdk-devel.x86_64
components,hadoop,hbase,yarn,mapred-app
hadoop_security,kerberos
hadoop_kerberos_domain,vagrant
hadoop_kerberos_realm,BIGTOP.ORG
hadoop_kerberos_kdc_server,bigtop1.vagrant
EOF

2.) Spin up the cluster.

$ ./startup.sh --cluster

3-1.) Get an error while formating namenode.

err: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Exec[namenode format]/returns: change from notrun to 0 failed: /bin/bash -c 'yes Y | hdfs namenode -format >> /var/lib/hadoop-hdfs/nn.format.log 2>&1' returned 1 instead of one of [0] at /tmp/vagrant-puppet-2/modules-0/hadoop/manifests/init.pp:361

3-2.) Get an error while starting up datanodes.

err: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Service[hadoop-hdfs-datanode]/ensure: change from stopped to running failed: Could not start Service[hadoop-hdfs-datanode]: Execution of '/sbin/service hadoop-hdfs-datanode start' returned 1:  at /tmp/vagrant-puppet-2/modules-0/hadoop/manifests/init.pp:158

Issue Links

• is blocked by: BIGTOP-1347 Support better entropy performance on vagrant VMs