Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1746

Introduce the concept of roles in bigtop cluster deployment

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.1.0
    • Component/s: deployment
    • Labels:

      Description

      Currently, during cluster deployment, puppet categorizes nodes as head_node, worker_nodes, gateway_nodes, standy_node based on user specified info. This functionality gives user control over picking up a particular node as head_node, standy_node, gateway_node and rest others as worker_nodes. But, I woulld like to have more fine-grained control on which deamons should run on which node. For example, I do not want to run namenode, datanode on the same node. This functionality can be introduced with the concept of roles. Each node can be assigned a set of role. For example, Node A can be assigned ["namenode", "resourcemanager"] roles. Node B can be assigned ["datanode", "nodemanager"] and Node C can be assigned ["nodemanager", "hadoop-client"]. Now, each node will only run the specified daemons. Prerequisite for this kind of deployment is that each node should be given the necessary configurations that it needs to know. For example, each datanode should know which is the namenode etc... This functionality will allow users to customize the cluster deployment according to their needs.

      1. BIGTOP-1746.patch
        41 kB
        vishnu gajendran
      2. BIGTOP-1746.patch
        41 kB
        vishnu gajendran
      3. BIGTOP-1746.patch
        41 kB
        vishnu gajendran
      4. BIGTOP-1746.patch
        36 kB
        vishnu gajendran
      5. BIGTOP-1746.patch
        36 kB
        vishnu gajendran

        Issue Links

          Activity

          Hide
          cos Konstantin Boudnik added a comment -

          Thanks for opening this up! I think it chimes in with the discussion we are currently having on BIGTOP-1702 and dev@ list starting from http://bit.ly/1BqBri4
          Would be great if you can chime in there !

          Show
          cos Konstantin Boudnik added a comment - Thanks for opening this up! I think it chimes in with the discussion we are currently having on BIGTOP-1702 and dev@ list starting from http://bit.ly/1BqBri4 Would be great if you can chime in there !
          Hide
          warwithin YoungWoo Kim added a comment -

          vishnu gajendran Good idea!

          FYI. if you are familiar with Ansible, you can try BIGTOP-1584.

          Show
          warwithin YoungWoo Kim added a comment - vishnu gajendran Good idea! FYI. if you are familiar with Ansible, you can try BIGTOP-1584 .
          Hide
          michaelweiser Michael Weiser added a comment -

          At least in the Puppet manifests there are only two places where the current role concept is implemented:

          • manifests/cluster.pp decides what daemons to put on which box
          • hieradata/bigtop/cluster.yaml governs what to put into their config files

          cluster.yaml can easily be adjusted and overridden so it doesn't force the concept of head node and frontend into the config files any more. So the main point of attack from my point of view is cluster.pp. Unfortunately it also implements some dependencies between modules and some add-on logic, such as running init-hdfs.sh. Basically I would suggest moving these dependencies into their respective modules and then just throwing away cluster.pp. After that, the classes could be included directly using hiera_include with the hiera lookup hierachy or possibly an ENC or facts governing which roles a machine has.

          I have a setup where I've basically done that. I have changes I was already planning to propose for merging that move dependencies mostly into the hadoop module. That would render cluster.pp quite empty already. I also have a concept for assigning roles to nodes via hiera. This, however, is a bit convoluted and would need streamlining for inclusion in mainline BigTop. In the most basic case, classes such as hadoop::namenode can just be assigned to nodes directly such as this:

          manifests/site.pp:

          hiera_include("classes")
          

          hiera.yaml:

          ---
          :yaml:
            :datadir: /etc/puppet/hieradata
          :hierarchy:
            - "node/%{::fqdn}"
            - site
            - bigtop/cluster
          

          hieradata/node/node1.do.main.yaml:

          ---
          classes:
            - hadoop::namenode
            - hadoop-zookeeper::server
          
          Show
          michaelweiser Michael Weiser added a comment - At least in the Puppet manifests there are only two places where the current role concept is implemented: manifests/cluster.pp decides what daemons to put on which box hieradata/bigtop/cluster.yaml governs what to put into their config files cluster.yaml can easily be adjusted and overridden so it doesn't force the concept of head node and frontend into the config files any more. So the main point of attack from my point of view is cluster.pp. Unfortunately it also implements some dependencies between modules and some add-on logic, such as running init-hdfs.sh. Basically I would suggest moving these dependencies into their respective modules and then just throwing away cluster.pp. After that, the classes could be included directly using hiera_include with the hiera lookup hierachy or possibly an ENC or facts governing which roles a machine has. I have a setup where I've basically done that. I have changes I was already planning to propose for merging that move dependencies mostly into the hadoop module. That would render cluster.pp quite empty already. I also have a concept for assigning roles to nodes via hiera. This, however, is a bit convoluted and would need streamlining for inclusion in mainline BigTop. In the most basic case, classes such as hadoop::namenode can just be assigned to nodes directly such as this: manifests/site.pp: hiera_include("classes") hiera.yaml: --- :yaml: :datadir: /etc/puppet/hieradata :hierarchy: - "node/%{::fqdn}" - site - bigtop/cluster hieradata/node/node1.do.main.yaml: --- classes: - hadoop::namenode - hadoop-zookeeper::server
          Hide
          rnp Richard Pelavin added a comment -

          An approach to look at is having a simple topology DSL say in yaml that the
          end user interacts with to specify the logical set of nodes, what daemons
          are on each node and how they are interconnected (for more complex things
          like ha or indicating how monitors, user authentication ,etc could plug
          in).. It would be easy then to write some simple code to "compile this into
          for example" hiera.yaml files or as an ENC or even both.

          I think this begs the question as to what type of end user is expected to
          use this; if it is a Puppet savvy end user then having them specify things
          in hiera would be clear, but if going after end users that are not well
          versed in.Puppet having a Bigtop topology dsl" may resonate better and can
          be simpler.

          Now, I have done much work around this area so would be happy to propose a
          starting point for a topology DSL if this approach makes sense. I can also
          flesh out a number of issues that could be addressed to see what priority
          people would give it.
          One issue for example is whether the topology just logically identifies
          nodes and groups of nodes (e.g., the set of slaves) and does not require ip
          or dns addresses to be assigned to them; this allows more sharable designs
          without locking users into what would be one team's deployment specific
          settings. It also facilitates the process where given a toplogy we spin up
          a set of nodes and in a late binding way attach the host addresses to the
          nodes and to the attributes for connecting between hosts.

          For the specific example of init-hdfs,sh if I am correctly guessing at what
          issue would be is that ideally want to only create directories for services
          that are being actually used and not create it for all directories or
          equivalently for the data driven way in which this is created you want to
          construct the description of directories to include as a function of what
          deamons are on the topology nodes. This is something I have tackled and can
          include this in a write up if interested,

          Show
          rnp Richard Pelavin added a comment - An approach to look at is having a simple topology DSL say in yaml that the end user interacts with to specify the logical set of nodes, what daemons are on each node and how they are interconnected (for more complex things like ha or indicating how monitors, user authentication ,etc could plug in).. It would be easy then to write some simple code to "compile this into for example" hiera.yaml files or as an ENC or even both. I think this begs the question as to what type of end user is expected to use this; if it is a Puppet savvy end user then having them specify things in hiera would be clear, but if going after end users that are not well versed in.Puppet having a Bigtop topology dsl" may resonate better and can be simpler. Now, I have done much work around this area so would be happy to propose a starting point for a topology DSL if this approach makes sense. I can also flesh out a number of issues that could be addressed to see what priority people would give it. One issue for example is whether the topology just logically identifies nodes and groups of nodes (e.g., the set of slaves) and does not require ip or dns addresses to be assigned to them; this allows more sharable designs without locking users into what would be one team's deployment specific settings. It also facilitates the process where given a toplogy we spin up a set of nodes and in a late binding way attach the host addresses to the nodes and to the attributes for connecting between hosts. For the specific example of init-hdfs,sh if I am correctly guessing at what issue would be is that ideally want to only create directories for services that are being actually used and not create it for all directories or equivalently for the data driven way in which this is created you want to construct the description of directories to include as a function of what deamons are on the topology nodes. This is something I have tackled and can include this in a write up if interested,
          Hide
          vishnu vishnu gajendran added a comment -

          Thanks for your comment Michael. I would like to read the patch you already worked on. would be great if you can share it. Thanks.

          Show
          vishnu vishnu gajendran added a comment - Thanks for your comment Michael. I would like to read the patch you already worked on. would be great if you can share it. Thanks.
          Hide
          vishnu vishnu gajendran added a comment -

          Thanks Richard for your comment. The model you are proposing looks more extensible and generic. Currently, dependencies between modules (like hbase depends on hdfs) is explicitly handled in bigtop cluster.pp file (or) as Michael wiesner commented on this issue above, we can handle these dependencies inside respective puppet modules. But, what you are proposing is that user should be able to explicitly define the dependencies (topology) between components. I feel this model will be very generic and difficult to handle. Also, currently, bigtop uses puppet in standalone mode. So, we cannot handle dependencies across nodes which is a limitation in scenarios like HA (for example, you need to ensure that all journal nodes are up and running before starting the namenode service). Please let me know your comments.

          Show
          vishnu vishnu gajendran added a comment - Thanks Richard for your comment. The model you are proposing looks more extensible and generic. Currently, dependencies between modules (like hbase depends on hdfs) is explicitly handled in bigtop cluster.pp file (or) as Michael wiesner commented on this issue above, we can handle these dependencies inside respective puppet modules. But, what you are proposing is that user should be able to explicitly define the dependencies (topology) between components. I feel this model will be very generic and difficult to handle. Also, currently, bigtop uses puppet in standalone mode. So, we cannot handle dependencies across nodes which is a limitation in scenarios like HA (for example, you need to ensure that all journal nodes are up and running before starting the namenode service). Please let me know your comments.
          Hide
          rnp Richard Pelavin added a comment - - edited

          Vishnu,

          Good comments. I think useful to divide into two issues:

          1) having a simple topology representation that is more intuitive and neutral; one that resonates with those who dont know hiera, vagrant, etc

          2) having such a representation that has the flexibility to capture a wide variety of topologies

          Think if 1 is implemented we could write simple code that "compiled" the topology into hiera/vagrant, etc and then power by existing bigtop instrumentation/mechanism.

          I think your comments are centered on '2'. The way I would handle this is as follows, which is predicated on whether there is a desire in bigtop to move in the direction of providing more sophisticated deployment and orchestration capabilities:

          An approach is to flesh out a rich topology that we would eventually want to instrument, but we could view this as an internal spec and then just expose/support a subset of the topologies that are instrumented currently by bigtop, such as a limited set of topologies that have a single master and set of slaves. So in a way the rich topology serves as a roadmap to future extensions and can help capture in advance how potential extensions to treat a richer set of topologies can be modularly folded in.

          Rich

          Show
          rnp Richard Pelavin added a comment - - edited Vishnu, Good comments. I think useful to divide into two issues: 1) having a simple topology representation that is more intuitive and neutral; one that resonates with those who dont know hiera, vagrant, etc 2) having such a representation that has the flexibility to capture a wide variety of topologies Think if 1 is implemented we could write simple code that "compiled" the topology into hiera/vagrant, etc and then power by existing bigtop instrumentation/mechanism. I think your comments are centered on '2'. The way I would handle this is as follows, which is predicated on whether there is a desire in bigtop to move in the direction of providing more sophisticated deployment and orchestration capabilities: An approach is to flesh out a rich topology that we would eventually want to instrument, but we could view this as an internal spec and then just expose/support a subset of the topologies that are instrumented currently by bigtop, such as a limited set of topologies that have a single master and set of slaves. So in a way the rich topology serves as a roadmap to future extensions and can help capture in advance how potential extensions to treat a richer set of topologies can be modularly folded in. Rich
          Hide
          vishnu vishnu gajendran added a comment -

          makes sense. We can decouple this as two separate issues. In this issue, we should track the second point and address the topology representation in a separate issue.

          Show
          vishnu vishnu gajendran added a comment - makes sense. We can decouple this as two separate issues. In this issue, we should track the second point and address the topology representation in a separate issue.
          Hide
          vishnu vishnu gajendran added a comment -

          I am working on this issue. will post a patch soon. I cannot assign this jira to me. Do I need to get special permission to assign jira?

          Show
          vishnu vishnu gajendran added a comment - I am working on this issue. will post a patch soon. I cannot assign this jira to me. Do I need to get special permission to assign jira?
          Hide
          vishnu vishnu gajendran added a comment -

          thanks Konstantin Boudnik for assigning it to me. I uploaded a patch for this issue. Appreciate the community comments on this. Thanks.

          Show
          vishnu vishnu gajendran added a comment - thanks Konstantin Boudnik for assigning it to me. I uploaded a patch for this issue. Appreciate the community comments on this. Thanks.
          Hide
          evans_ye Evans Ye added a comment -

          Hi vishnu gajendran, I'd like to thank you for devoting lots of energy in this great feature we're eager to have. Please allow me to chime in with my personal opinion. I've briefly read through the patch and it seems that currently we need to drop different configuration files that indicate the roles of node on different nodes. I understand that in this patch you'd like to target on the second point, while the first point should be another story. But what we've done in this patch somehow directly related to the implementation when we dealing with the first point.
          So, here's what I can see to have a topology representation implemented after your patch:
          we can get FQDN from facter and use FQDN to determine what roles a node needs to be provisioned. For instance, we can have site.yaml like this:

          node1.do.main
            - hadoop::namenode
            - hadoop-zookeeper::server
          node2.do.main
            - hadoop::datanode
            - hadoop-zookeeper::server
          node3.do.main
            - hbase::master
            - hadoop-zookeeper::server
          

          when doing puppet apply, each puppet agent first matches FQDN in configuration file and fetch its roles, then proceed to the deployment based on role definitions just as what your patch does.

          I'm not proposing to modify your patch. Instead, I just want to make sure that this can be applied on top of your patch so that we can have this in and address the topology representation in another JIRA.

          Show
          evans_ye Evans Ye added a comment - Hi vishnu gajendran , I'd like to thank you for devoting lots of energy in this great feature we're eager to have. Please allow me to chime in with my personal opinion. I've briefly read through the patch and it seems that currently we need to drop different configuration files that indicate the roles of node on different nodes. I understand that in this patch you'd like to target on the second point, while the first point should be another story. But what we've done in this patch somehow directly related to the implementation when we dealing with the first point. So, here's what I can see to have a topology representation implemented after your patch: we can get FQDN from facter and use FQDN to determine what roles a node needs to be provisioned. For instance, we can have site.yaml like this: node1. do .main - hadoop::namenode - hadoop-zookeeper::server node2. do .main - hadoop::datanode - hadoop-zookeeper::server node3. do .main - hbase::master - hadoop-zookeeper::server when doing puppet apply, each puppet agent first matches FQDN in configuration file and fetch its roles, then proceed to the deployment based on role definitions just as what your patch does. I'm not proposing to modify your patch. Instead, I just want to make sure that this can be applied on top of your patch so that we can have this in and address the topology representation in another JIRA.
          Hide
          michaelweiser Michael Weiser added a comment -

          vishnu gajendran: I've had a quick glance at the patch. This looks like awesome stuff! Unfortunately I won't have time to try it out this week but I'll test it ASAP. Sorry as well for not making my patches available but they don't seem to overlap with what you've done anyway. A couple of quick questions before I dive into it deeper:

          Am I right in assuming that you tried to retain compatibility with the current components concept? Have you thought about being more radical and dropping components support in favour of perhaps a couple of template $fqdn.yamls that implement the same thing? I think this would be awesome in terms of code cleanup and should be just as easily integrateable with vagrant and other frameworks: Instead of setting bigtop::hadoop_head_node in site.yaml one just copies the appropriate template to head.node.do.main.yaml. I think we can be radical here since with the introduction of hiera the interface changed substantially quite recently anyway.

          Why have you decided on a two-level role layout like zookeeper subrole client? Couldn't we just direcly include hadoop-zookeeper::client from $roles_map or directly from hiera?

          Without the two-level roles, would we still need the deploy subclasses in each module? They mostly seem like 1:1 redirections to me. The only exception in gridgain, which IMO was mistakenly made a defined type and could without fallout be changed into an includeable class.

          Minor niggle: If the main class is (correctly) called bigtop_util, the directory should be bigtop_util as well.

          Evans Ye: With hiera we have get away from thinking in terms of one configuration file. Since hiera is a configuration database with a configurable lookup path, you can do stuff like I've sketched out in comment-14354586. So in stead of doing a role mapping in site.yaml, we can just do role assignments in node-specific node.do.main.yamls.

          Show
          michaelweiser Michael Weiser added a comment - vishnu gajendran : I've had a quick glance at the patch. This looks like awesome stuff! Unfortunately I won't have time to try it out this week but I'll test it ASAP. Sorry as well for not making my patches available but they don't seem to overlap with what you've done anyway. A couple of quick questions before I dive into it deeper: Am I right in assuming that you tried to retain compatibility with the current components concept? Have you thought about being more radical and dropping components support in favour of perhaps a couple of template $fqdn.yamls that implement the same thing? I think this would be awesome in terms of code cleanup and should be just as easily integrateable with vagrant and other frameworks: Instead of setting bigtop::hadoop_head_node in site.yaml one just copies the appropriate template to head.node.do.main.yaml. I think we can be radical here since with the introduction of hiera the interface changed substantially quite recently anyway. Why have you decided on a two-level role layout like zookeeper subrole client? Couldn't we just direcly include hadoop-zookeeper::client from $roles_map or directly from hiera? Without the two-level roles, would we still need the deploy subclasses in each module? They mostly seem like 1:1 redirections to me. The only exception in gridgain, which IMO was mistakenly made a defined type and could without fallout be changed into an includeable class. Minor niggle: If the main class is (correctly) called bigtop_util, the directory should be bigtop_util as well. Evans Ye : With hiera we have get away from thinking in terms of one configuration file. Since hiera is a configuration database with a configurable lookup path, you can do stuff like I've sketched out in comment-14354586 . So in stead of doing a role mapping in site.yaml, we can just do role assignments in node-specific node.do.main.yamls.
          Hide
          vishnu vishnu gajendran added a comment -

          Hi Evans, thank you for reviewing my patch. You are correct. I have targeted this patch to add roles as additional parameter for bigtop cluster deployment. For your question related to having node (FQDN) specific configuration, I like the idea which Michael weiser mentioned in his below comments.

          Show
          vishnu vishnu gajendran added a comment - Hi Evans, thank you for reviewing my patch. You are correct. I have targeted this patch to add roles as additional parameter for bigtop cluster deployment. For your question related to having node (FQDN) specific configuration, I like the idea which Michael weiser mentioned in his below comments.
          Hide
          vishnu vishnu gajendran added a comment -

          Hi Michael, thank you for reviewing my patch. yes, I have tried to retain the concept of components in this patch. I think existing configuration mechanism (specifying just head node and components list) will be handy if user wants to deploy a "test" single or multinode cluster without dealing with much of specific configurations.

          On your question regarding two-level roles, our thinking is its better not to expose the internal implementation details (i.e class and sub-class). Plus, not every module need to support roles (for utility classes like bigtop-utils etc...). Also, I have included explicit "deploy" class for each module and this class will be reponsible for dealing with dependencies between services (during installation) when roles are specified.

          Also, I will change the class name for bigtop-utils module.

          Show
          vishnu vishnu gajendran added a comment - Hi Michael, thank you for reviewing my patch. yes, I have tried to retain the concept of components in this patch. I think existing configuration mechanism (specifying just head node and components list) will be handy if user wants to deploy a "test" single or multinode cluster without dealing with much of specific configurations. On your question regarding two-level roles, our thinking is its better not to expose the internal implementation details (i.e class and sub-class). Plus, not every module need to support roles (for utility classes like bigtop-utils etc...). Also, I have included explicit "deploy" class for each module and this class will be reponsible for dealing with dependencies between services (during installation) when roles are specified. Also, I will change the class name for bigtop-utils module.
          Hide
          vishnu vishnu gajendran added a comment -

          fixed bigtop-utils module class name as Michael mentioned and uploaded a new patch.

          Show
          vishnu vishnu gajendran added a comment - fixed bigtop-utils module class name as Michael mentioned and uploaded a new patch.
          Hide
          evans_ye Evans Ye added a comment - - edited

          Hi Michael Weiser and vishnu gajendran I've read through some hiera document and I'm now completely agree with you. Thank you both for responding to my question.

          I give the patch a shot and I found that there's an issue occurred which I believe you have already mentioned in above comments.

          Here's my testing scenario on a two node cluster

          • bigtop1 container:
            bigtop::roles_enabled: true
            bigtop::roles:
             - namenode
             - datanode
            

            *bigtop2 container:

            bigtop::roles_enabled: true
            bigtop::roles:
             - resourcemanager
             - nodemanager
            

            *bigtop3 container:

            bigtop::roles_enabled: true
            bigtop::roles:
             - datanode
             - nodemanager
            

          bigtop1 can be successfully deployed while bigtop2 stopped at:

          Error: Could not find resource 'Class[Hadoop::Init_hdfs]' for relationship on 'Class[Hadoop::Resourcemanager]' on node bigtop2.docker
          

          Since we're using puppet in masterless mode, RM never know when hdfs has been initialed.
          From my limited brain I can only think of something like doing Exec to check a status file on HDFS which serve as a go signal. Once hdfs inited, RM can proceed to deploy.
          Or, we can simply remove the order settings and let user do puppet apply in particular order regarding to its topology. Take mine topology as an example:
          do puppet apply on namenode node --> do puppet apply resourcemanager node --> do puppet apply on slaves...
          Thoughts?

          Show
          evans_ye Evans Ye added a comment - - edited Hi Michael Weiser and vishnu gajendran I've read through some hiera document and I'm now completely agree with you. Thank you both for responding to my question. I give the patch a shot and I found that there's an issue occurred which I believe you have already mentioned in above comments. Here's my testing scenario on a two node cluster bigtop1 container: bigtop::roles_enabled: true bigtop::roles: - namenode - datanode *bigtop2 container: bigtop::roles_enabled: true bigtop::roles: - resourcemanager - nodemanager *bigtop3 container: bigtop::roles_enabled: true bigtop::roles: - datanode - nodemanager bigtop1 can be successfully deployed while bigtop2 stopped at: Error: Could not find resource ' Class [Hadoop::Init_hdfs]' for relationship on ' Class [Hadoop::Resourcemanager]' on node bigtop2.docker Since we're using puppet in masterless mode, RM never know when hdfs has been initialed. From my limited brain I can only think of something like doing Exec to check a status file on HDFS which serve as a go signal. Once hdfs inited, RM can proceed to deploy. Or, we can simply remove the order settings and let user do puppet apply in particular order regarding to its topology. Take mine topology as an example: do puppet apply on namenode node --> do puppet apply resourcemanager node --> do puppet apply on slaves... Thoughts?
          Hide
          cos Konstantin Boudnik added a comment -

          I would not want to have a remove execs involved as it will further complicates the deployment mechanism, IMO.

          Interestingly enough, over the years I have never seen a situation where HDFS-dependent components is deployed before HDFS is done. So, I presume our Puppet delivers exactly what it suppose to. Why this order question has raised now (sorry if I missed some context)? Thanks

          Show
          cos Konstantin Boudnik added a comment - I would not want to have a remove execs involved as it will further complicates the deployment mechanism, IMO. Interestingly enough, over the years I have never seen a situation where HDFS-dependent components is deployed before HDFS is done. So, I presume our Puppet delivers exactly what it suppose to. Why this order question has raised now (sorry if I missed some context)? Thanks
          Hide
          vishnu vishnu gajendran added a comment -

          Hi Evans, Thanks for pointing out the issue. To give more context on this issue, this is happening because, there is a dependency set between Init_hdfs class and Hadoop::Resourcemanager class. But, Init_hdfs class is actually executed in the node where namenode is installed. In your node topology, namenode and Resourcemanager runs on different nodes. So, Init_hdfs class is not executed on the node where Resourcemanager runs although there is a dependency set between Class['Hadoop::Init_hdfs'] -> Class['Hadoop::Resourcemanager'] which will cause puppet to throw an error like one you mentioned.

          This is something which I would like to address with https://issues.apache.org/jira/browse/BIGTOP-1772 issue. With this, rather than having a single init_hdfs script, every module is responsible for creating respective folders in hdfs.

          Show
          vishnu vishnu gajendran added a comment - Hi Evans, Thanks for pointing out the issue. To give more context on this issue, this is happening because, there is a dependency set between Init_hdfs class and Hadoop::Resourcemanager class. But, Init_hdfs class is actually executed in the node where namenode is installed. In your node topology, namenode and Resourcemanager runs on different nodes. So, Init_hdfs class is not executed on the node where Resourcemanager runs although there is a dependency set between Class ['Hadoop::Init_hdfs'] -> Class ['Hadoop::Resourcemanager'] which will cause puppet to throw an error like one you mentioned. This is something which I would like to address with https://issues.apache.org/jira/browse/BIGTOP-1772 issue. With this, rather than having a single init_hdfs script, every module is responsible for creating respective folders in hdfs.
          Hide
          cos Konstantin Boudnik added a comment -

          So, Init_hdfs class is not executed on the node where Resourcemanager runs

          init-hdfs shouldn't be executed on RM node - only on HDFS headnode. So, it seems that puppet recipe does exactly what it suppose to do, doesn't it?

          Show
          cos Konstantin Boudnik added a comment - So, Init_hdfs class is not executed on the node where Resourcemanager runs init-hdfs shouldn't be executed on RM node - only on HDFS headnode. So, it seems that puppet recipe does exactly what it suppose to do, doesn't it?
          Hide
          vishnu vishnu gajendran added a comment -

          yes, you are correct. This is an expected behaviour. But, at the same time, you do not want to start ResourceManager (resourcemanager runs on a different node than the hdfs head node) before Init_hdfs is run.

          Show
          vishnu vishnu gajendran added a comment - yes, you are correct. This is an expected behaviour. But, at the same time, you do not want to start ResourceManager (resourcemanager runs on a different node than the hdfs head node) before Init_hdfs is run.
          Hide
          cos Konstantin Boudnik added a comment -

          well, right now the way to deal with something like that is to re-run the recipes if there're any errors detected. Adding an artificial dependency - which, by the way, will require init-hcfs operation to be idempotent - seems like an overshot to me.

          Show
          cos Konstantin Boudnik added a comment - well, right now the way to deal with something like that is to re-run the recipes if there're any errors detected. Adding an artificial dependency - which, by the way, will require init-hcfs operation to be idempotent - seems like an overshot to me.
          Hide
          vishnu vishnu gajendran added a comment -

          okay. so, to fix this quickly, I can add one more condition to say that set Class['Hadoop::Init_hdfs'] -> Class['Hadoop::Resourcemanager'] dependency on the node only if "namenode" role is specified on the same node. This will make sure that dependency is set correctly and Init_hdfs runs only on namenode node.

          Show
          vishnu vishnu gajendran added a comment - okay. so, to fix this quickly, I can add one more condition to say that set Class ['Hadoop::Init_hdfs'] -> Class ['Hadoop::Resourcemanager'] dependency on the node only if "namenode" role is specified on the same node. This will make sure that dependency is set correctly and Init_hdfs runs only on namenode node.
          Hide
          evans_ye Evans Ye added a comment -

          Hi vishnu gajendran I think your idea for my question is good.
          However, here's what I found more for the patch
          Even though I've specified resourcemanager on bigtop2 container, the yarn-site.xml is still configured to hadoop_head_node, hence the service cannot connected.
          I need to update following configuration in hieradata/bigtop/cluster.yaml to have correct host set in yarn-site.xml.

          #hadoop::common_yarn::hadoop_rm_host: "%{hiera('bigtop::hadoop_head_node')}"
          hadoop::common_yarn::hadoop_rm_host: "bigtop2.docker"
          

          Here we have two options:

          • Parse specified roles in bigtop::roles and then update the hadoop_rm_host. I don't have an idea how this is going to be implemented since each node only have its own role settings... So we don't have a way to know resourcemanager host on other hosts.
          • Specify topology by binding roles directly to the host, for example:
            hadoop::common_yarn::hadoop_rm_host: "bigtop2.docker"
            

            Then we can determine the role by checking the host array. For example, a logic like this:

            if ($fqdn in $hadoop_rm_host) {
              include hadoop::resourcemanager
            }
            

            I'd like to ping Michael Weiser to join the discussion. You have more knowledge on the concept of hadoop::common_yarn::hadoop_rm_host sort of stuff

          Show
          evans_ye Evans Ye added a comment - Hi vishnu gajendran I think your idea for my question is good. However, here's what I found more for the patch Even though I've specified resourcemanager on bigtop2 container, the yarn-site.xml is still configured to hadoop_head_node , hence the service cannot connected. I need to update following configuration in hieradata/bigtop/cluster.yaml to have correct host set in yarn-site.xml . #hadoop::common_yarn::hadoop_rm_host: "%{hiera('bigtop::hadoop_head_node')}" hadoop::common_yarn::hadoop_rm_host: "bigtop2.docker" Here we have two options: Parse specified roles in bigtop::roles and then update the hadoop_rm_host . I don't have an idea how this is going to be implemented since each node only have its own role settings... So we don't have a way to know resourcemanager host on other hosts. Specify topology by binding roles directly to the host, for example: hadoop::common_yarn::hadoop_rm_host: "bigtop2.docker" Then we can determine the role by checking the host array. For example, a logic like this: if ($fqdn in $hadoop_rm_host) { include hadoop::resourcemanager } I'd like to ping Michael Weiser to join the discussion. You have more knowledge on the concept of hadoop::common_yarn::hadoop_rm_host sort of stuff
          Hide
          vishnu vishnu gajendran added a comment -

          yes, the assumption (which I pointed in the jira description) is

          "... Prerequisite for this kind of deployment is that each node should be given the necessary configurations that it needs to know. For example, each datanode should know which is the namenode etc..."

          The limitation here is that we do not use the puppet master-agent model (or something similar) for bigtop cluster deployment. So, one way or other, we need to supply necessary configs to all nodes.

          Thanks Evans for taking your time testing my patch!

          Show
          vishnu vishnu gajendran added a comment - yes, the assumption (which I pointed in the jira description) is "... Prerequisite for this kind of deployment is that each node should be given the necessary configurations that it needs to know. For example, each datanode should know which is the namenode etc..." The limitation here is that we do not use the puppet master-agent model (or something similar) for bigtop cluster deployment. So, one way or other, we need to supply necessary configs to all nodes. Thanks Evans for taking your time testing my patch!
          Hide
          evans_ye Evans Ye added a comment -

          No prob Vishnu, I'd like to get this awesome stuff in. Since the change is quite big so we take times doing review and testing.
          The problem here is users need to configure these two configuration correctly to get the role successfully provisioned:

          • bigtop::roles: [resourcemanager]
          • hadoop::common_yarn::hadoop_rm_host: "bigtop2.docker"

          In addition, by introducing bigtop::roles, we need to have a set of predefined roles that used to be matched in each module's init.pp:

          if ("resourcemanager" in $roles)
          

          So we have to define resourcemanager, hbase-master, hbase-server, etc as tags of roles.

          What about we try the second option I mentioned above? In that case we simply let user to configure FQDNs in host variables:

          hadoop_namenode_host: bigtop1.docker
          hadoop_datanode_hosts: [bigtop2.docker, bigtop3.docker]
          hadoop_rm_host: bigtop2.docker
          

          And then update if ("resourcemanager" in $roles) to if ($fqdn == $hadoop_rm_host) in init.pp. Then we're pretty much good to go. We don't have to remember what tags to be specified, and we only let user specify one configuration to get what they need.
          Thoughts.

          Show
          evans_ye Evans Ye added a comment - No prob Vishnu, I'd like to get this awesome stuff in. Since the change is quite big so we take times doing review and testing. The problem here is users need to configure these two configuration correctly to get the role successfully provisioned: bigtop::roles: [resourcemanager] hadoop::common_yarn::hadoop_rm_host: "bigtop2.docker" In addition, by introducing bigtop::roles, we need to have a set of predefined roles that used to be matched in each module's init.pp: if ( "resourcemanager" in $roles) So we have to define resourcemanager, hbase-master, hbase-server, etc as tags of roles. What about we try the second option I mentioned above? In that case we simply let user to configure FQDNs in host variables: hadoop_namenode_host: bigtop1.docker hadoop_datanode_hosts: [bigtop2.docker, bigtop3.docker] hadoop_rm_host: bigtop2.docker And then update if ("resourcemanager" in $roles) to if ($fqdn == $hadoop_rm_host) in init.pp. Then we're pretty much good to go. We don't have to remember what tags to be specified, and we only let user specify one configuration to get what they need. Thoughts.
          Hide
          rleidle Rob Leidle added a comment -

          To me the variables for defining the list of namenode hosts (should be an array as it would be useful for HA), list of variables for datanode, etc, in the end is going to get translated into a role. In addition, it means that we need to define these variables for every app. Consider the case of spark, we would need to define the following for the basic use cases (spark-on-yarn, standalone mode, a restful job server for receiving requests, and running the zeppelin web interface:

          spark_on_yarn_hosts: [ ]
          spark_master_hosts: [ ]
          spark_worker_hosts: [ ]
          spark_job_server_hosts: [ ]
          zeppelin_hosts: [ ]

          It seems to me that these convenience variables can play fine with roles and in fact they are a layer on top of roles as they are implicitly defining roles. I would propose the following:

          • Remove the variable for enabling roles. They should always be enabled.
          • Roles array remains, it defines what roles are provisioned for the current node that puppet apply is running on.
          • When the puppet apply process starts it can take a look at these 'special' variables such as "hadoop_namenode_hosts". If the fqdn of the current node is in that list it will add "namenode" to the list of roles if it is not present already.

          Thoughts?

          Show
          rleidle Rob Leidle added a comment - To me the variables for defining the list of namenode hosts (should be an array as it would be useful for HA), list of variables for datanode, etc, in the end is going to get translated into a role. In addition, it means that we need to define these variables for every app. Consider the case of spark, we would need to define the following for the basic use cases (spark-on-yarn, standalone mode, a restful job server for receiving requests, and running the zeppelin web interface: spark_on_yarn_hosts: [ ] spark_master_hosts: [ ] spark_worker_hosts: [ ] spark_job_server_hosts: [ ] zeppelin_hosts: [ ] It seems to me that these convenience variables can play fine with roles and in fact they are a layer on top of roles as they are implicitly defining roles. I would propose the following: Remove the variable for enabling roles. They should always be enabled. Roles array remains, it defines what roles are provisioned for the current node that puppet apply is running on. When the puppet apply process starts it can take a look at these 'special' variables such as "hadoop_namenode_hosts". If the fqdn of the current node is in that list it will add "namenode" to the list of roles if it is not present already. Thoughts?
          Hide
          vishnu vishnu gajendran added a comment -

          Evans, your point is very valid that user might input inconsistent configuration to bigtop since we are using two different configs: one is list of roles and the other is master daemon host FQDN for respective applications. Your suggestion of using FQDN to determine roles is also a valid point. But, when I think of node provisioning, to provision a new node to a existing cluster with lets say just hdfs datanode process, it would be very convenient to specify "datanode" as a role for the new node rather than adding the new node FQDN to the list of datanode hosts. I am talking from a user experience perspective. What do you say?

          Show
          vishnu vishnu gajendran added a comment - Evans, your point is very valid that user might input inconsistent configuration to bigtop since we are using two different configs: one is list of roles and the other is master daemon host FQDN for respective applications. Your suggestion of using FQDN to determine roles is also a valid point. But, when I think of node provisioning, to provision a new node to a existing cluster with lets say just hdfs datanode process, it would be very convenient to specify "datanode" as a role for the new node rather than adding the new node FQDN to the list of datanode hosts. I am talking from a user experience perspective. What do you say?
          Hide
          evans_ye Evans Ye added a comment -

          In general, I think Rob Leidle and I are sharing the same point.
          What we currently have by applying the patch is users may think of that they only need to specify roles to get all the things done. While things may become that they get a cluster which is not correctly configured and need to figure out the mis-configured part by themselves. My thought on having better user experience is we should avoid troubleshooting happened on user end as much as we can.

          Having good user experience is always the goal we're pursuing. If we can have the logic to automatically transform roles definition into host settings in *-site.xml, then I vote for your proposal since it's easier to understand. (I can't figure the implementation out, maybe someone who is smarter?)

          I'm proposing the FQDN as roles design because I think this might be relatively easy to accomplish by doing sightly updates on your patch, and which does not have too bad UX. To me I think that adding datanode based on this design won't be to hard. Just have the site.yaml updated from hadoop_datanode_hosts: [bigtop2.docker, bigtop3.docker] to hadoop_datanode_hosts: [bigtop2.docker, bigtop3.docker, bigtop4.docker].
          And it's all the same when adding a nodemanager to the cluster (the EMR case). While the purely roles design needs to not only have role specified but also have hadoop_rm_host configured.
          In production we usually have master nodes like namenode, hmaster, resourcemanager, spark-master, etc configured on different nodes to ensure the stability. So it's often that hadoop_rm_host does not pointing to the hadoop_head_node.

          Show
          evans_ye Evans Ye added a comment - In general, I think Rob Leidle and I are sharing the same point. What we currently have by applying the patch is users may think of that they only need to specify roles to get all the things done. While things may become that they get a cluster which is not correctly configured and need to figure out the mis-configured part by themselves. My thought on having better user experience is we should avoid troubleshooting happened on user end as much as we can. Having good user experience is always the goal we're pursuing. If we can have the logic to automatically transform roles definition into host settings in *-site.xml, then I vote for your proposal since it's easier to understand. (I can't figure the implementation out, maybe someone who is smarter?) I'm proposing the FQDN as roles design because I think this might be relatively easy to accomplish by doing sightly updates on your patch, and which does not have too bad UX. To me I think that adding datanode based on this design won't be to hard. Just have the site.yaml updated from hadoop_datanode_hosts: [bigtop2.docker, bigtop3.docker] to hadoop_datanode_hosts: [bigtop2.docker, bigtop3.docker, bigtop4.docker] . And it's all the same when adding a nodemanager to the cluster (the EMR case). While the purely roles design needs to not only have role specified but also have hadoop_rm_host configured. In production we usually have master nodes like namenode, hmaster, resourcemanager, spark-master, etc configured on different nodes to ensure the stability. So it's often that hadoop_rm_host does not pointing to the hadoop_head_node.
          Hide
          cos Konstantin Boudnik added a comment -

          Is it just me or this feels like an attempt to get closer to the master-based Puppet deployment, but without actual master?

          Show
          cos Konstantin Boudnik added a comment - Is it just me or this feels like an attempt to get closer to the master-based Puppet deployment, but without actual master?
          Hide
          vishnu vishnu gajendran added a comment -

          Evans, sure. makes sense. Just to clarify, What Rob Leidle says is different from yours. He says, lets retain the label (roles) for each component. Lets also add a function which will map

          hadoop_namenode_host: bigtop1.docker
          hadoop_datanode_hosts: [bigtop2.docker, bigtop3.docker]
          hadoop_rm_host: bigtop2.docker

          to respective roles. we then take union of both the specified roles and the roles derived from the hosts list mentioned above. Here, label field is just for convenience (for example, to install hive-client etc...)

          Show
          vishnu vishnu gajendran added a comment - Evans, sure. makes sense. Just to clarify, What Rob Leidle says is different from yours. He says, lets retain the label (roles) for each component. Lets also add a function which will map hadoop_namenode_host: bigtop1.docker hadoop_datanode_hosts: [bigtop2.docker, bigtop3.docker] hadoop_rm_host: bigtop2.docker to respective roles. we then take union of both the specified roles and the roles derived from the hosts list mentioned above. Here, label field is just for convenience (for example, to install hive-client etc...)
          Hide
          vishnu vishnu gajendran added a comment -

          Hi Konstantin Boudnik, you are correct. the primary limitation is that there is no mechanism for nodes to share information. Its either that we provide all information to all nodes (like what Evans is suggesting) or supply only needed information to a node (which I was suggesting earlier).

          Show
          vishnu vishnu gajendran added a comment - Hi Konstantin Boudnik, you are correct. the primary limitation is that there is no mechanism for nodes to share information. Its either that we provide all information to all nodes (like what Evans is suggesting) or supply only needed information to a node (which I was suggesting earlier).
          Hide
          evans_ye Evans Ye added a comment -

          Yes you're absolutely right. I don't have opinion on that cause that's more related to the implementation rather than the user experience. Therefore I think you as the patch developer would know which way is better. I'd be happy to help on committing this. So, let me know your thoughts, or just ping me if you're ready.

          Show
          evans_ye Evans Ye added a comment - Yes you're absolutely right. I don't have opinion on that cause that's more related to the implementation rather than the user experience. Therefore I think you as the patch developer would know which way is better. I'd be happy to help on committing this. So, let me know your thoughts, or just ping me if you're ready.
          Hide
          vishnu vishnu gajendran added a comment -

          Hi Evans, I gave little more thoughts on this patch. I think having labels for specifying roles offers two benefits over the idea you proposed:

          1. A clear abstraction on what to install and run on a node rather than relying on specific configuration settings (like namenode host(s), resourcemanager host, datanode hosts etc...). This means that we decouple puppet deployment mechanism from component specific configuration settings (configuration settings may vary depending on the mode of installation for each component, for example, spark on yarn vs standard spark installation). But, to address the configuration mismatch you encountered earlier, we can add a validation check at runtime in puppet to ensure that configuration settings (like resourcemanager host) matches the FQDN of node for which resourcemanager role is assigned.
          2. I like the idea of assigning roles to a host without knowing any specific info about the host (like FQDN)

          what do you think about this solution?

          Show
          vishnu vishnu gajendran added a comment - Hi Evans, I gave little more thoughts on this patch. I think having labels for specifying roles offers two benefits over the idea you proposed: 1. A clear abstraction on what to install and run on a node rather than relying on specific configuration settings (like namenode host(s), resourcemanager host, datanode hosts etc...). This means that we decouple puppet deployment mechanism from component specific configuration settings (configuration settings may vary depending on the mode of installation for each component, for example, spark on yarn vs standard spark installation). But, to address the configuration mismatch you encountered earlier, we can add a validation check at runtime in puppet to ensure that configuration settings (like resourcemanager host) matches the FQDN of node for which resourcemanager role is assigned. 2. I like the idea of assigning roles to a host without knowing any specific info about the host (like FQDN) what do you think about this solution?
          Hide
          evans_ye Evans Ye added a comment - - edited

          Hey vishnu gajendran, thanks for getting back to this JIRA.
          If I were getting it right, the first point looks compelling to me.
          Let's say, we have spark-standalone-master, spark-yarn-master and spark-mesos-master tags, with these tags we can specify how we'd like to deploy spark on spark_master_host, although we currently only supports, IIRC, standalone mode, it seems many components can have such different mode to deploy. So it's good to have the design being flexible.
          Instead of validating the FQDN setting in puppet, is it possible that we just assign it to the correct FQDN at runtime according to hiera FQDN-tags configuration?

          Show
          evans_ye Evans Ye added a comment - - edited Hey vishnu gajendran , thanks for getting back to this JIRA. If I were getting it right, the first point looks compelling to me. Let's say, we have spark-standalone-master , spark-yarn-master and spark-mesos-master tags, with these tags we can specify how we'd like to deploy spark on spark_master_host , although we currently only supports, IIRC, standalone mode, it seems many components can have such different mode to deploy. So it's good to have the design being flexible. Instead of validating the FQDN setting in puppet, is it possible that we just assign it to the correct FQDN at runtime according to hiera FQDN-tags configuration?
          Hide
          vishnu vishnu gajendran added a comment -

          you mean auto-populate spark_master_host based on assigned roles. thats possible. But, the problem is we are dealing with master-less puppet where each node does not know about other node. So, FQDN for lets say namenode or any master daemon needs to be communicated using hiera configuration to all other worker nodes.

          Show
          vishnu vishnu gajendran added a comment - you mean auto-populate spark_master_host based on assigned roles. thats possible. But, the problem is we are dealing with master-less puppet where each node does not know about other node. So, FQDN for lets say namenode or any master daemon needs to be communicated using hiera configuration to all other worker nodes.
          Hide
          evans_ye Evans Ye added a comment -

          Yup I was thinking in that, for example:

          cat FQDN.yaml
          role:
          - spark-standalone-master
          

          This kind of role definitions will be shipped to all the nodes, so those nodes might be able to get FQDN of spark master and fill it into spark_master_host.

          However, getting back to the JIRA, I think it's enough to only have a validation right now. We can also document down what set of configuration users need to modify, then we're fine.
          Would you mind to give me a heads up on the purpose of gemfile and gemfile.lock?
          I don't have knowledge on it yet

          Show
          evans_ye Evans Ye added a comment - Yup I was thinking in that, for example: cat FQDN.yaml role: - spark-standalone-master This kind of role definitions will be shipped to all the nodes, so those nodes might be able to get FQDN of spark master and fill it into spark_master_host. However, getting back to the JIRA, I think it's enough to only have a validation right now. We can also document down what set of configuration users need to modify, then we're fine. Would you mind to give me a heads up on the purpose of gemfile and gemfile.lock? I don't have knowledge on it yet
          Hide
          vishnu vishnu gajendran added a comment -

          sure. I will work on that. the purpose of gemfile and gemfile.lock is to define the some of the dependent ruby modules used in ruby scripts get_roles.rb and the unit test script which I wrote for get_roles method (get_roles_spec.rb file). Looking at the Gemfile now, I have defined many other modules which are not used by the scripts. I will remove those unused ones in my next revision of the patch.

          Show
          vishnu vishnu gajendran added a comment - sure. I will work on that. the purpose of gemfile and gemfile.lock is to define the some of the dependent ruby modules used in ruby scripts get_roles.rb and the unit test script which I wrote for get_roles method (get_roles_spec.rb file). Looking at the Gemfile now, I have defined many other modules which are not used by the scripts. I will remove those unused ones in my next revision of the patch.
          Hide
          evans_ye Evans Ye added a comment -

          OK, thanks for the explanation.
          Would you like to ship this feature in 1.0 release?
          Anyhow, feel free to ping me for the review when you're ready.

          Show
          evans_ye Evans Ye added a comment - OK, thanks for the explanation. Would you like to ship this feature in 1.0 release? Anyhow, feel free to ping me for the review when you're ready.
          Hide
          vishnu vishnu gajendran added a comment -

          Is there a cutoff date for the release? I am new to apache, not familiar with the release model..

          Show
          vishnu vishnu gajendran added a comment - Is there a cutoff date for the release? I am new to apache, not familiar with the release model..
          Hide
          evans_ye Evans Ye added a comment -

          Once we reach all green status across build, deploy, and test, then we should avoid any patch that can interfere the correctness of release code base except bug fixes.
          Unfortunately, we're not at that moment yet.
          This JIRA is on 1.0 fix list. So, I can work with you to get it in if you'd like to, but I think fixing this in next release would be ok, too.

          Show
          evans_ye Evans Ye added a comment - Once we reach all green status across build, deploy, and test, then we should avoid any patch that can interfere the correctness of release code base except bug fixes. Unfortunately, we're not at that moment yet. This JIRA is on 1.0 fix list. So, I can work with you to get it in if you'd like to, but I think fixing this in next release would be ok, too.
          Hide
          cos Konstantin Boudnik added a comment -

          What Evans says.

          Show
          cos Konstantin Boudnik added a comment - What Evans says.
          Hide
          cos Konstantin Boudnik added a comment -

          What Evans says.

          Show
          cos Konstantin Boudnik added a comment - What Evans says.
          Hide
          rnp Richard Pelavin added a comment -

          I had a question/comments about extensibility. My understanding of patch is that in effect you need to tag the nodes that the cluster is being deployed on as being one of these four

          • Standby
          • Master
          • Gateway
          • Worker

          and that way you are tagging it is by giving fqdns for all but worker and assuming if don't match any of these three addresses then it is worker

          Simple extension I am thinking about adding a ganglia server on a seperate node. Assuming that only one category can be given by omission so I would also need a fqdn for ganglia server (or have fqdns for all the workers)

          Now a generalization that is something to potentially tackle in a subsequent jira is allowing different tagging mechanisms. For example in a ec2 scenario through cloud init it is easy to posit a file on each spun up node that indicates whether the node is master, standby, etc. it is also easy to define a facter variable that determines this. I think this can be easily handled by generalizing the test that determines what node type it is; that is, rather than hard coded tests against the fqdn having a puppet function that takes the tagging method being used as a parameter which can be passed down by hiera

          Rich

          Show
          rnp Richard Pelavin added a comment - I had a question/comments about extensibility. My understanding of patch is that in effect you need to tag the nodes that the cluster is being deployed on as being one of these four Standby Master Gateway Worker and that way you are tagging it is by giving fqdns for all but worker and assuming if don't match any of these three addresses then it is worker Simple extension I am thinking about adding a ganglia server on a seperate node. Assuming that only one category can be given by omission so I would also need a fqdn for ganglia server (or have fqdns for all the workers) Now a generalization that is something to potentially tackle in a subsequent jira is allowing different tagging mechanisms. For example in a ec2 scenario through cloud init it is easy to posit a file on each spun up node that indicates whether the node is master, standby, etc. it is also easy to define a facter variable that determines this. I think this can be easily handled by generalizing the test that determines what node type it is; that is, rather than hard coded tests against the fqdn having a puppet function that takes the tagging method being used as a parameter which can be passed down by hiera Rich
          Hide
          vishnu vishnu gajendran added a comment -

          Thanks for the info. Lets not include this for 1.0 release since this is a major change. will fix the patch release version to indicate this.

          Show
          vishnu vishnu gajendran added a comment - Thanks for the info. Lets not include this for 1.0 release since this is a major change. will fix the patch release version to indicate this.
          Hide
          vishnu vishnu gajendran added a comment -

          Thanks for the info. Lets not include this for 1.0 release since this is a major change. will fix the patch release version to indicate this.

          Show
          vishnu vishnu gajendran added a comment - Thanks for the info. Lets not include this for 1.0 release since this is a major change. will fix the patch release version to indicate this.
          Hide
          evans_ye Evans Ye added a comment -

          Hey vishnu gajendran I saw your new patch, and would like to review it. But sorry I might not have cycles for this before 1.0 released.
          I'm back on this after the release. Thank you for keep working on this.

          Show
          evans_ye Evans Ye added a comment - Hey vishnu gajendran I saw your new patch, and would like to review it. But sorry I might not have cycles for this before 1.0 released. I'm back on this after the release. Thank you for keep working on this.
          Hide
          vishnu vishnu gajendran added a comment -

          sure. thank you for your help.

          Show
          vishnu vishnu gajendran added a comment - sure. thank you for your help.
          Hide
          vishnu vishnu gajendran added a comment -

          Hi Rich,

          Apologies for late response. The tags (master, worker, gateway, standby) are used only to maintain the backward compatibility with existing code (which is determine who is master and worker based on master fqdn and install appropriate components on each node). When roles are enabled and assigned to each node, these tags and the logic which determines what components to install based on fqdn is not applicable. Its upto the user to define what to install on a node by assigning roles to each node. In your case, for ganglia server, you can define a role called "ganglia-server" for ganglia application and assign that role to the node where you want to install. Please let me know if I am missing something.

          Show
          vishnu vishnu gajendran added a comment - Hi Rich, Apologies for late response. The tags (master, worker, gateway, standby) are used only to maintain the backward compatibility with existing code (which is determine who is master and worker based on master fqdn and install appropriate components on each node). When roles are enabled and assigned to each node, these tags and the logic which determines what components to install based on fqdn is not applicable. Its upto the user to define what to install on a node by assigning roles to each node. In your case, for ganglia server, you can define a role called "ganglia-server" for ganglia application and assign that role to the node where you want to install. Please let me know if I am missing something.
          Hide
          rnp Richard Pelavin added a comment -

          Thanks Vishnu,

          That clarified things for me. Approach looks real good.

          Rich

          Show
          rnp Richard Pelavin added a comment - Thanks Vishnu, That clarified things for me. Approach looks real good. Rich
          Hide
          evans_ye Evans Ye added a comment -

          Hi vishnu gajendran I'm sorry to get back for the review so late.
          I did a review on your patch, but since the patch can't apply directly to the current master, I can not run tests for it. A question I'm curious about: why the hbase-master role implies zookeeper server installation?

          23	    if ("hbase-master" in $roles) {
          24	      include hadoop::init_hdfs
          25	      include hadoop-hbase::master
          26	      include hadoop-zookeeper::server
          27	      Class['Hadoop::Init_hdfs'] -> Class['Hadoop-hbase::Master']
          28	    }
          

          I'm imaging that there should be a dependency to install zookeeper services somewhere else in the cluster, for example, Class['Hadoop::Zookeeper'] -> Class['Hadoop-hbase::Master'] and then we can start to install hbase-master.
          Am I getting it right?

          The 1.0 is about to release, so If you're still interesting in contributing the feature, I'm being able to work with you to get it in.

          Show
          evans_ye Evans Ye added a comment - Hi vishnu gajendran I'm sorry to get back for the review so late. I did a review on your patch, but since the patch can't apply directly to the current master, I can not run tests for it. A question I'm curious about: why the hbase-master role implies zookeeper server installation? 23 if ( "hbase-master" in $roles) { 24 include hadoop::init_hdfs 25 include hadoop-hbase::master 26 include hadoop-zookeeper::server 27 Class ['Hadoop::Init_hdfs'] -> Class ['Hadoop-hbase::Master'] 28 } I'm imaging that there should be a dependency to install zookeeper services somewhere else in the cluster, for example, Class ['Hadoop::Zookeeper'] -> Class ['Hadoop-hbase::Master'] and then we can start to install hbase-master. Am I getting it right? The 1.0 is about to release, so If you're still interesting in contributing the feature, I'm being able to work with you to get it in.
          Hide
          vishnu vishnu gajendran added a comment -

          Hi Evans, Thanks for reviewing the patch. The reason to include hadoop::Zookeeper::Server here is to maintain the existing logic. If you see the current code,

          if ($all or "hbase" in $components)

          { include hadoop-zookeeper::server }

          is defined in cluster.pp file. But, I agree to your point that hbase module need not install the zookeeper::server component by itself. Rather, it can check if zookeeper::server role is present, if yes, then it can set Class['Hadoop::Zookeeper'] -> Class['Hadoop-hbase::Master'] dependency. Let me know if I misunderstood what you said.

          Sure, I would like to contribute this back. I am currently working on merging my patch with latest code. I will test it and post a new patch in may be couple of days.

          Show
          vishnu vishnu gajendran added a comment - Hi Evans, Thanks for reviewing the patch. The reason to include hadoop::Zookeeper::Server here is to maintain the existing logic. If you see the current code, if ($all or "hbase" in $components) { include hadoop-zookeeper::server } is defined in cluster.pp file. But, I agree to your point that hbase module need not install the zookeeper::server component by itself. Rather, it can check if zookeeper::server role is present, if yes, then it can set Class ['Hadoop::Zookeeper'] -> Class ['Hadoop-hbase::Master'] dependency. Let me know if I misunderstood what you said. Sure, I would like to contribute this back. I am currently working on merging my patch with latest code. I will test it and post a new patch in may be couple of days.
          Hide
          vishnu vishnu gajendran added a comment -

          Hi Evans, please find the latest patch attached. There is a change in functionality with regards to tez which is described below. Also, I am not very familiar with tez components (tez-client), so, correct me if I am wrong.

          Currently, bigtop automagically configures hadoop and hive to use tez as execution engine when you specify tez as a component in bigtop::components list. With introduction of roles, this functionality has changed. This is because there is only one role for tez which is "tez-client" which may or may not be installed on the same machine as hadoop/hive. So, its upto to the user to configure hadoop/hive to use tez as execution engine by specifying the following configurations.

          1. to enable tez in hadoop, uncomment the lines below
          2. hadoop::common::use_tez: true
          3. hadoop::common_mapred_app::mapreduce_framework_name: "yarn-tez"
          1. to enable tez in hive, uncomment the lines below
          2. hadoop-hive::client::use_tez: true

          I understand this is not very convenient for users given that users has to explicitly specify these configurations. So, looking forward for your comments/suggestions. Thanks.

          Show
          vishnu vishnu gajendran added a comment - Hi Evans, please find the latest patch attached. There is a change in functionality with regards to tez which is described below. Also, I am not very familiar with tez components (tez-client), so, correct me if I am wrong. Currently, bigtop automagically configures hadoop and hive to use tez as execution engine when you specify tez as a component in bigtop::components list. With introduction of roles, this functionality has changed. This is because there is only one role for tez which is "tez-client" which may or may not be installed on the same machine as hadoop/hive. So, its upto to the user to configure hadoop/hive to use tez as execution engine by specifying the following configurations. to enable tez in hadoop, uncomment the lines below hadoop::common::use_tez: true hadoop::common_mapred_app::mapreduce_framework_name: "yarn-tez" to enable tez in hive, uncomment the lines below hadoop-hive::client::use_tez: true I understand this is not very convenient for users given that users has to explicitly specify these configurations. So, looking forward for your comments/suggestions. Thanks.
          Hide
          evans_ye Evans Ye added a comment - - edited

          Hey, vishnu gajendran. First of all, thanks for staying with me on this. If you find something need to discuss, just post your thought here. We'd love to discuss. There's no need to provide patch in hurry.
          My two cents, might not be the best:

          • Instead of using the configuration hadoop-hive::client::use_tez: true, which is to change the hive execution engine value to tez, how about we explicitly specify it as hadoop-hive::client::exec_engine: tez? Just like what you did for mapreduce_framework_name.
          • It looks like the value of hadoop::common::use_tez is used to decide whether or not to add tez libs into HADOOP_CLASSPATH, which is fine. But this brings up another issue I can recall that we've discussed in mailing list
            We haven't reach a conclusion yet, but the truth is, we're using different practice for different packages. Take iginte as an example, it directly smylink iginte libs into hadoop/libs so that adding classpath is no longer needed:
            [root@bigtop1 lib]# ll /usr/lib/hadoop/lib |grep ignite
            lrwxrwxrwx 1 root root      43 Aug 17 16:15 ignite-core.jar -> /usr/lib/ignite-hadoop/libs/ignite-core.jar
            lrwxrwxrwx 1 root root      59 Aug 17 16:15 ignite-hadoop.jar -> /usr/lib/ignite-hadoop/libs/ignite-hadoop/ignite-hadoop.jar
            

          At this moment I think we should decide which practice we're going for. Does anyone have a quick answer how the other deployment system do?
          Anyone also has opinions to the patch? Probably Michael Weiser, Richard Pelavin, Rob Leidle can add some color.

          Show
          evans_ye Evans Ye added a comment - - edited Hey, vishnu gajendran . First of all, thanks for staying with me on this. If you find something need to discuss, just post your thought here. We'd love to discuss. There's no need to provide patch in hurry. My two cents, might not be the best: Instead of using the configuration hadoop-hive::client::use_tez: true , which is to change the hive execution engine value to tez, how about we explicitly specify it as hadoop-hive::client::exec_engine: tez ? Just like what you did for mapreduce_framework_name . It looks like the value of hadoop::common::use_tez is used to decide whether or not to add tez libs into HADOOP_CLASSPATH, which is fine. But this brings up another issue I can recall that we've discussed in mailing list We haven't reach a conclusion yet, but the truth is, we're using different practice for different packages. Take iginte as an example, it directly smylink iginte libs into hadoop/libs so that adding classpath is no longer needed: [root@bigtop1 lib]# ll /usr/lib/hadoop/lib |grep ignite lrwxrwxrwx 1 root root 43 Aug 17 16:15 ignite-core.jar -> /usr/lib/ignite-hadoop/libs/ignite-core.jar lrwxrwxrwx 1 root root 59 Aug 17 16:15 ignite-hadoop.jar -> /usr/lib/ignite-hadoop/libs/ignite-hadoop/ignite-hadoop.jar At this moment I think we should decide which practice we're going for. Does anyone have a quick answer how the other deployment system do? Anyone also has opinions to the patch? Probably Michael Weiser , Richard Pelavin , Rob Leidle can add some color.
          Hide
          rleidle Rob Leidle added a comment -

          My gut instinct is that symlinking into the hadoop lib is the wrong way to do it. It should be the other way around. Applications should symlink the hadoop libs into their classpath folder if necessary. This is to avoid the situation where an application needs the Hadoop libraries in the classpath but does not want the ignite libraries in the classpath.

          If they need to be part of the Hadoop CLASSPATH then their puppet provisioning recipes should modify the appropriate settings and environment files to make sure that happens.

          Show
          rleidle Rob Leidle added a comment - My gut instinct is that symlinking into the hadoop lib is the wrong way to do it. It should be the other way around. Applications should symlink the hadoop libs into their classpath folder if necessary. This is to avoid the situation where an application needs the Hadoop libraries in the classpath but does not want the ignite libraries in the classpath. If they need to be part of the Hadoop CLASSPATH then their puppet provisioning recipes should modify the appropriate settings and environment files to make sure that happens.
          Hide
          evans_ye Evans Ye added a comment -

          I think you make a good point here. Not doing symlink is to avoid a simple MR job to have other unused jars included in its classpath. What's your take on the role definition design?

          Show
          evans_ye Evans Ye added a comment - I think you make a good point here. Not doing symlink is to avoid a simple MR job to have other unused jars included in its classpath. What's your take on the role definition design?
          Hide
          rleidle Rob Leidle added a comment -

          Agree with the exec_engine setting. Better to have a field that accepts any execution engine than a collection of boolean settings with complicated behavior.

          Show
          rleidle Rob Leidle added a comment - Agree with the exec_engine setting. Better to have a field that accepts any execution engine than a collection of boolean settings with complicated behavior.
          Hide
          evans_ye Evans Ye added a comment - - edited

          vishnu gajendran do you think the setting of exec_engine making sense?
          OTOH, I did review and testing on the patch and discovered that tez does not work properly because the tez lib hasn't been uploaded to HDFS. I take a slightly deep look into this and found a dependency is missing in the patch:

          • original code:
              if ($all or "tez" in $components) {
                include tez::client
                Class['tez::client'] -> Exec<| title == "init hdfs" |>
              }
            
          • the patch:
              tez => {
                client => ["tez-client"],
              },
            

          The dependency setting Class['tez::client'] -> Exec<| title == "init hdfs" |> is for components which would like to upload libs to HDFS can wait for HDFS to be ready to avoid upload failure. The same logic has been implemented in oozie installation as well. Would you please add the logic back in your patch?
          I think the design here is good enough to get in. We don't need a perfect solution at the first place. If there's no other thing broken and you think the exec_engine thing make sense, then please upload a new patch with the fixes. We're close to the finished line.

          Show
          evans_ye Evans Ye added a comment - - edited vishnu gajendran do you think the setting of exec_engine making sense? OTOH, I did review and testing on the patch and discovered that tez does not work properly because the tez lib hasn't been uploaded to HDFS. I take a slightly deep look into this and found a dependency is missing in the patch: original code: if ($all or "tez" in $components) { include tez::client Class ['tez::client'] -> Exec<| title == "init hdfs" |> } the patch: tez => { client => [ "tez-client" ], }, The dependency setting Class ['tez::client'] -> Exec<| title == "init hdfs" |> is for components which would like to upload libs to HDFS can wait for HDFS to be ready to avoid upload failure. The same logic has been implemented in oozie installation as well. Would you please add the logic back in your patch? I think the design here is good enough to get in. We don't need a perfect solution at the first place. If there's no other thing broken and you think the exec_engine thing make sense, then please upload a new patch with the fixes. We're close to the finished line.
          Hide
          vishnu vishnu gajendran added a comment -

          Hi Evans, Thanks a lot for testing the patch. I will fix the bug you mentioned. I think the exec_engine makes sense and simplifies things little bit on the user end. I will incorporate these changes and post a new patch soon.

          Show
          vishnu vishnu gajendran added a comment - Hi Evans, Thanks a lot for testing the patch. I will fix the bug you mentioned. I think the exec_engine makes sense and simplifies things little bit on the user end. I will incorporate these changes and post a new patch soon.
          Hide
          vishnu vishnu gajendran added a comment -

          Hi Evans, I have a question on the dependency you mentioned. If what you said is correct, then the dependency should be other way around

          Exec<| title == "init hdfs" |> -> Class['tez::client']

          can you please confirm?

          Show
          vishnu vishnu gajendran added a comment - Hi Evans, I have a question on the dependency you mentioned. If what you said is correct, then the dependency should be other way around Exec<| title == "init hdfs" |> -> Class ['tez::client'] can you please confirm?
          Hide
          evans_ye Evans Ye added a comment -

          Sorry, my comment was wrong. Ignore what I said in previous comment. the dependency setting is correct.
          The reason why we setup the dependency is actually because init hdfs will upload necessary libs onto HDFS (tez libs, oozie libs), so we should make sure those libraries available before we do upload.

          Show
          evans_ye Evans Ye added a comment - Sorry, my comment was wrong. Ignore what I said in previous comment. the dependency setting is correct. The reason why we setup the dependency is actually because init hdfs will upload necessary libs onto HDFS (tez libs, oozie libs), so we should make sure those libraries available before we do upload.
          Hide
          vishnu vishnu gajendran added a comment -

          Hi Evans, I have attached the patch with the changes we discussed earlier. Please let me know your comments. Thanks.

          Show
          vishnu vishnu gajendran added a comment - Hi Evans, I have attached the patch with the changes we discussed earlier. Please let me know your comments. Thanks.
          Hide
          evans_ye Evans Ye added a comment -

          I've tested your patch and confirm that the two items are addressed in the latest patch. However, I'm trying to go further and deploy a cluster using role definitions. I haven't figure it out yet. Can you give me a hint or an example to show how to do that?
          OTOH, It would be great if you can open up another JIRA to update the puppet README explaining how to leverage this awesome feature.

          Show
          evans_ye Evans Ye added a comment - I've tested your patch and confirm that the two items are addressed in the latest patch. However, I'm trying to go further and deploy a cluster using role definitions. I haven't figure it out yet. Can you give me a hint or an example to show how to do that? OTOH, It would be great if you can open up another JIRA to update the puppet README explaining how to leverage this awesome feature.
          Hide
          evans_ye Evans Ye added a comment - - edited

          Hey vishnu gajendran I figured out how this works. Adding one line into hiera.yaml to get config by hostname:

          root@bigtop2 puppet]# cat hiera.yaml
          ---
          :yaml:
            :datadir: /etc/puppet/hieradata
          :hierarchy:
            - "bigtop/%{fqdn}"
            - site
            - "bigtop/%{hadoop_hiera_ha_path}"
            - bigtop/cluster
          

          Then we can have host specific role definitions in the following arrangement:

          hieradata/
          - bigtop/
            - bigtop1.docker.yaml
            - bigtop2.docker.yaml
            - bigtop3.docker.yaml
          

          and then ship the hieradata directory as configurations across the cluster

          Is this what you designed for?

          Show
          evans_ye Evans Ye added a comment - - edited Hey vishnu gajendran I figured out how this works. Adding one line into hiera.yaml to get config by hostname: root@bigtop2 puppet]# cat hiera.yaml --- :yaml: :datadir: /etc/puppet/hieradata :hierarchy: - "bigtop/%{fqdn}" - site - "bigtop/%{hadoop_hiera_ha_path}" - bigtop/cluster Then we can have host specific role definitions in the following arrangement: hieradata/ - bigtop/ - bigtop1.docker.yaml - bigtop2.docker.yaml - bigtop3.docker.yaml and then ship the hieradata directory as configurations across the cluster Is this what you designed for?
          Hide
          vishnu vishnu gajendran added a comment -

          Hi Evans, Sorry for the delay. Yes, thats exactly how it works.

          Show
          vishnu vishnu gajendran added a comment - Hi Evans, Sorry for the delay. Yes, thats exactly how it works.
          Hide
          evans_ye Evans Ye added a comment -

          Finally, with repeatedly testing and reviews, here's my +1 to this great new feature.
          Most of the feature I've tested are remain working, while the newly added role definition also works without a problem.
          If no objection I'll commit this later.

          Show
          evans_ye Evans Ye added a comment - Finally, with repeatedly testing and reviews, here's my +1 to this great new feature. Most of the feature I've tested are remain working, while the newly added role definition also works without a problem. If no objection I'll commit this later.
          Hide
          evans_ye Evans Ye added a comment -

          And committed !
          Thanks for your relentless will to contribute this vishnu gajendran. The work actually takes half a year to be in. Now, with the whole new role definition frame being settle, we can start the iteration to polish it in fast pace.

          Show
          evans_ye Evans Ye added a comment - And committed ! Thanks for your relentless will to contribute this vishnu gajendran . The work actually takes half a year to be in. Now, with the whole new role definition frame being settle, we can start the iteration to polish it in fast pace.
          Hide
          vishnu vishnu gajendran added a comment -

          Thanks a lot Evans for great suggestions and helping me in testing the patch.

          Show
          vishnu vishnu gajendran added a comment - Thanks a lot Evans for great suggestions and helping me in testing the patch.
          Hide
          tomzeng Tom Zeng added a comment -

          Great, thanks vishnu gajendran and Evans Ye

          Show
          tomzeng Tom Zeng added a comment - Great, thanks vishnu gajendran and Evans Ye

            People

            • Assignee:
              vishnu vishnu gajendran
              Reporter:
              vishnu vishnu gajendran
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development