Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1336

Puppet recipes failed to deploy kerberos enabled hadoop cluster

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: deployment
    • Labels:
      None

      Description

      Here are some missing dependency setting in our puppet recipes in order to get kerberos enabled on the hadoop cluster.

      The first one is that kerberos principal for hdfs user hasn't been created before formatting namenode, which cause the namenode formatting process failed.

      The second one is that /etc/default/hadoop-hdfs-datanode doesn't get ready before starting up datanodes and results in datanodes failed to startup.
      The datanode error log:

      2014-06-16 15:10:10,711 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
      java.lang.RuntimeException: Cannot start secure cluster without privileged resources.
      

      Here's the reproduce steps using vagrant-puppet:
      1.) Enable kerberos on the hadoop cluster.

      $ vim bigtop-deploy/vm/vagrant-puppet/provision.sh
      

      Add kerberos definitions.

      cat > /bigtop-puppet/config/site.csv << EOF
      hadoop_head_node,$1
      hadoop_storage_dirs,/data/1,/data/2
      bigtop_yumrepo_uri,http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64
      jdk_package_name,java-1.7.0-openjdk-devel.x86_64
      components,hadoop,hbase,yarn,mapred-app
      hadoop_security,kerberos
      hadoop_kerberos_domain,vagrant
      hadoop_kerberos_realm,BIGTOP.ORG
      hadoop_kerberos_kdc_server,bigtop1.vagrant
      EOF
      

      2.) Spin up the cluster.

      $ ./startup.sh --cluster
      

      3-1.) Get an error while formating namenode.

      err: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Exec[namenode format]/returns: change from notrun to 0 failed: /bin/bash -c 'yes Y | hdfs namenode -format >> /var/lib/hadoop-hdfs/nn.format.log 2>&1' returned 1 instead of one of [0] at /tmp/vagrant-puppet-2/modules-0/hadoop/manifests/init.pp:361
      

      3-2.) Get an error while starting up datanodes.

      err: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Service[hadoop-hdfs-datanode]/ensure: change from stopped to running failed: Could not start Service[hadoop-hdfs-datanode]: Execution of '/sbin/service hadoop-hdfs-datanode start' returned 1:  at /tmp/vagrant-puppet-2/modules-0/hadoop/manifests/init.pp:158
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                evans_ye Evans Ye
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: