Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2466

HBase puppet scripts co-deploy ZK and Datanode along with HBase

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: hbase
    • Labels:
      None

      Description

      Deploying the only HBase roles fails because it is expected to also co-deploy a datanode. Similarly, the puppet HBase scripts deploy a Zookeeper along with HBase.

      In deployments where ZK and HDFS are already provided on other nodes you end up needlessly deploying services along with HBase and waste resources.

      We should have a way to deploy only HBase (without auxiliary services) from the puppet scripts.

        Issue Links

          Activity

          Hide
          cos Konstantin Boudnik added a comment -

          Basically, you're saying about having more finer grained roles, right?

          Show
          cos Konstantin Boudnik added a comment - Basically, you're saying about having more finer grained roles, right?
          Hide
          ktsakalozos Konstantinos Tsakalozos added a comment -

          Yes, you could describe it like that.

          My understanding is that when you deploy a role you get exactly what the role describes. So if I need HBase master only I would deploy "hbase-master" role. If I want to have a full blown HBase deployment then we should have a component named HBase that would deploy all the roles needed (hbase-master, hbase-server, hbase-client, zookeeper, whatever datanode roles we need, etc...). The way hbase-master role is right now you can only deploy it if hadoop-client role is deployed as well; and as soon hbase-master gets deployed you realise zookeeper is also running which is a service you didn't ask for. I believe we will run into trouble if we try to deploy HBase on a node where zookeeper or a datanode is already running.

          Show
          ktsakalozos Konstantinos Tsakalozos added a comment - Yes, you could describe it like that. My understanding is that when you deploy a role you get exactly what the role describes. So if I need HBase master only I would deploy "hbase-master" role. If I want to have a full blown HBase deployment then we should have a component named HBase that would deploy all the roles needed (hbase-master, hbase-server, hbase-client, zookeeper, whatever datanode roles we need, etc...). The way hbase-master role is right now you can only deploy it if hadoop-client role is deployed as well; and as soon hbase-master gets deployed you realise zookeeper is also running which is a service you didn't ask for. I believe we will run into trouble if we try to deploy HBase on a node where zookeeper or a datanode is already running.
          Hide
          ktsakalozos Konstantinos Tsakalozos added a comment - - edited

          For completeness let me show how deploying only hbase roles fails.

          Starting with this site.yaml:

          root@jackal-local-machine-4:/home/ubuntu/bigtop.release/bigtop-1.1.0# cat  bigtop-deploy/puppet/hieradata/site.yaml 
          bigtop::bigtop_repo_uri: http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/ubuntu/trusty/x86_64
          bigtop::hadoop_head_node: jackal-local-machine-5
          bigtop::jdk_preinstalled: true
          bigtop::roles:
          - hbase-client
          - hbase-master
          - hbase-server
          bigtop::roles_enabled: true
          hadoop::common_hdfs::hadoop_namenode_host: jackal-local-machine-5
          hadoop::common_hdfs::namenode_datanode_registration_ip_hostname_check: true
          hadoop::common_mapred_app::jobtracker_host: jackal-local-machine-6
          hadoop::common_mapred_app::mapreduce_jobhistory_host: jackal-local-machine-6
          hadoop::common_yarn::hadoop_ps_host: jackal-local-machine-6
          hadoop::common_yarn::hadoop_rm_host: jackal-local-machine-6
          hadoop::common_yarn::yarn_nodemanager_vmem_check_enabled: false
          hadoop::hadoop_storage_dirs:
          - /data/1
          - /data/2
          hadoop_hbase::common_config::zookeeper_quorum: 10.0.3.165
          

          And doing a:

          puppet apply -d --modulepath="bigtop-deploy/puppet/modules:/etc/puppet/modules" bigtop-deploy/puppet/manifests/site.pp
          

          You get his error:

          Error: Could not find dependency Package[hadoop-hdfs] for Exec[init hdfs] at /home/ubuntu/bigtop.release/bigtop-1.1.0/bigtop-deploy/puppet/modules/hadoop/manifests/init.pp:87
          
          Show
          ktsakalozos Konstantinos Tsakalozos added a comment - - edited For completeness let me show how deploying only hbase roles fails. Starting with this site.yaml: root@jackal-local-machine-4:/home/ubuntu/bigtop.release/bigtop-1.1.0# cat bigtop-deploy/puppet/hieradata/site.yaml bigtop::bigtop_repo_uri: http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/ubuntu/trusty/x86_64 bigtop::hadoop_head_node: jackal-local-machine-5 bigtop::jdk_preinstalled: true bigtop::roles: - hbase-client - hbase-master - hbase-server bigtop::roles_enabled: true hadoop::common_hdfs::hadoop_namenode_host: jackal-local-machine-5 hadoop::common_hdfs::namenode_datanode_registration_ip_hostname_check: true hadoop::common_mapred_app::jobtracker_host: jackal-local-machine-6 hadoop::common_mapred_app::mapreduce_jobhistory_host: jackal-local-machine-6 hadoop::common_yarn::hadoop_ps_host: jackal-local-machine-6 hadoop::common_yarn::hadoop_rm_host: jackal-local-machine-6 hadoop::common_yarn::yarn_nodemanager_vmem_check_enabled: false hadoop::hadoop_storage_dirs: - /data/1 - /data/2 hadoop_hbase::common_config::zookeeper_quorum: 10.0.3.165 And doing a: puppet apply -d --modulepath="bigtop-deploy/puppet/modules:/etc/puppet/modules" bigtop-deploy/puppet/manifests/site.pp You get his error: Error: Could not find dependency Package[hadoop-hdfs] for Exec[init hdfs] at /home/ubuntu/bigtop.release/bigtop-1.1.0/bigtop-deploy/puppet/modules/hadoop/manifests/init.pp:87
          Hide
          ktsakalozos Konstantinos Tsakalozos added a comment -

          Patch to allow auxiliary services to be skipped, with in mind not to brake the behaviour we have until now.

          Show
          ktsakalozos Konstantinos Tsakalozos added a comment - Patch to allow auxiliary services to be skipped, with in mind not to brake the behaviour we have until now.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user ktsakalozos opened a pull request:

          https://github.com/apache/bigtop/pull/124

          BIGTOP-2466: HBase puppet scripts co-deploy ZK and Datanode along with HBase

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/juju-solutions/bigtop BIGTOP-2466

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/124.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #124


          commit 4d143150a034bdce8a965e3e707c824174361f5f
          Author: Konstantinos Tsakalozos <konstantinos.tsakalozos@canonical.com>
          Date: 2016-06-03T07:22:46Z

          BIGTOP-2466: HBase puppet scripts co-deploy ZK and Datanode along with HBase


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user ktsakalozos opened a pull request: https://github.com/apache/bigtop/pull/124 BIGTOP-2466 : HBase puppet scripts co-deploy ZK and Datanode along with HBase You can merge this pull request into a Git repository by running: $ git pull https://github.com/juju-solutions/bigtop BIGTOP-2466 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/124.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #124 commit 4d143150a034bdce8a965e3e707c824174361f5f Author: Konstantinos Tsakalozos <konstantinos.tsakalozos@canonical.com> Date: 2016-06-03T07:22:46Z BIGTOP-2466 : HBase puppet scripts co-deploy ZK and Datanode along with HBase
          Hide
          rvs Roman Shaposhnik added a comment -

          Konstantinos Tsakalozos I a bit confused. Your failure example mentions hadoop-hdfs as the failed dependency, but your PR is focused on making zookeeper conditional. Am I missing something?

          Show
          rvs Roman Shaposhnik added a comment - Konstantinos Tsakalozos I a bit confused. Your failure example mentions hadoop-hdfs as the failed dependency, but your PR is focused on making zookeeper conditional. Am I missing something?
          Hide
          ktsakalozos Konstantinos Tsakalozos added a comment -

          There are two issues that are addressed in this patch:

          1. Zookeeper was always deployed along with HBase. This patch does not change the default behaviour, but also allows for skipping the Zookeeper service so that HBase can relate to another Zookeeper deployment.

          2. The deployment of only HBase services was failing (as reported in previous comment) with:

          Error: Could not find dependency Package[hadoop-hdfs]
          

          this is fixed with an easily missed:

          include hadoop::common_hdfs
          

          Thank you for your time.

          Show
          ktsakalozos Konstantinos Tsakalozos added a comment - There are two issues that are addressed in this patch: 1. Zookeeper was always deployed along with HBase. This patch does not change the default behaviour, but also allows for skipping the Zookeeper service so that HBase can relate to another Zookeeper deployment. 2. The deployment of only HBase services was failing (as reported in previous comment) with: Error: Could not find dependency Package[hadoop-hdfs] this is fixed with an easily missed: include hadoop::common_hdfs Thank you for your time.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/124

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/124

            People

            • Assignee:
              ktsakalozos Konstantinos Tsakalozos
              Reporter:
              ktsakalozos Konstantinos Tsakalozos
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development