Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: deployment
    • Labels:
      None
    1. BIGTOP-2123.patch
      10 kB
      Konstantin Boudnik

      Issue Links

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        The patch for new recipes. It isn't finished yet, as even when the role for zeppelin is enabled the packages aren't getting installed nor configured. Jonathan Kelly, would you mind taking a look into this?

        Show
        cos Konstantin Boudnik added a comment - The patch for new recipes. It isn't finished yet, as even when the role for zeppelin is enabled the packages aren't getting installed nor configured. Jonathan Kelly , would you mind taking a look into this?
        Hide
        jonathak Jonathan Kelly added a comment -

        Thanks for splitting up my original patch for me, Konstantin Boudnik. When you enable the role for zeppelin, are you using "zeppelin-server" or just "zeppelin"? As you can see in the deploy class of bigtop-deploy/puppet/modules/zeppelin/manifests/init.pp, it should be "zeppelin-server".

        Show
        jonathak Jonathan Kelly added a comment - Thanks for splitting up my original patch for me, Konstantin Boudnik . When you enable the role for zeppelin, are you using "zeppelin-server" or just "zeppelin"? As you can see in the deploy class of bigtop-deploy/puppet/modules/zeppelin/manifests/init.pp, it should be "zeppelin-server".
        Hide
        cos Konstantin Boudnik added a comment -

        Yeah, here's the snippet of my site.yaml

        bigtop::roles:
          - namenode
          - datanode
          - ignite-hadoop
          - spark-master
          - spark-worker
          - zeppelin-server
        

        and it still doesn't get deployed with {[apply}}.
        Another part I really don't like is about current approach to the installation. Looks like the requirement to use bigtop::role to deploy a component goes against a simple need to adda component hadoop_cluster_node::cluster_components list. Let's fix it.

        Show
        cos Konstantin Boudnik added a comment - Yeah, here's the snippet of my site.yaml bigtop::roles: - namenode - datanode - ignite-hadoop - spark-master - spark-worker - zeppelin-server and it still doesn't get deployed with {[apply}}. Another part I really don't like is about current approach to the installation. Looks like the requirement to use bigtop::role to deploy a component goes against a simple need to adda component hadoop_cluster_node::cluster_components list. Let's fix it.
        Hide
        cos Konstantin Boudnik added a comment -

        Well, in order to get something installed by Bigtop-wide recipes you need to add your new modules to bigtop-deploy/puppet/manifests/cluster.pp like here

        --- bigtop-deploy/puppet/manifests/cluster.pp
        +++ bigtop-deploy/puppet/manifests/cluster.pp
        @@ -102,6 +102,9 @@ $roles_map = {
           ycsb => {
             client => ["ycsb-client"],
           },
        +  zeppelin => {
        +    server => ["zeppelin"],
        +  },
         }
         
         class hadoop_cluster_node (
        @@ -166,7 +169,8 @@ class node_with_roles ($roles = hiera("bigtop::roles")) inherits hadoop_cluster_
             "tachyon",
             "tez",
             "ycsb",
        -    "kerberos"
        +    "kerberos",
        +    "zeppelin"
           ]
         
           deploy_module { $modules:
        

        But even then the new recipe is buggy

        Error: Invalid parameter overrides on File[/etc/zeppelin/conf/zeppelin-env.sh] at /ws/bigtop-deploy/puppet/modules/zeppelin/manifests/init.pp:39 on node bigtop1.docker
        

        Please fix these so we can move forward.

        Show
        cos Konstantin Boudnik added a comment - Well, in order to get something installed by Bigtop-wide recipes you need to add your new modules to bigtop-deploy/puppet/manifests/cluster.pp like here --- bigtop-deploy/puppet/manifests/cluster.pp +++ bigtop-deploy/puppet/manifests/cluster.pp @@ -102,6 +102,9 @@ $roles_map = { ycsb => { client => [ "ycsb-client" ], }, + zeppelin => { + server => [ "zeppelin" ], + }, } class hadoop_cluster_node ( @@ -166,7 +169,8 @@ class node_with_roles ($roles = hiera( "bigtop::roles" )) inherits hadoop_cluster_ "tachyon" , "tez" , "ycsb" , - "kerberos" + "kerberos" , + "zeppelin" ] deploy_module { $modules: But even then the new recipe is buggy Error: Invalid parameter overrides on File[/etc/zeppelin/conf/zeppelin-env.sh] at /ws/bigtop-deploy/puppet/modules/zeppelin/manifests/init.pp:39 on node bigtop1.docker Please fix these so we can move forward.
        Hide
        cos Konstantin Boudnik added a comment -

        Default web-port for zeppelin server is in the conflict with spark master ui port

        Show
        cos Konstantin Boudnik added a comment - Default web-port for zeppelin server is in the conflict with spark master ui port
        Hide
        cos Konstantin Boudnik added a comment -

        This

               $spark_master_url = 'yarn-client',
        

        Assume that we are always running Spark over Yarn. Which isn't the case by default. Please fix it accordingly.

        Show
        cos Konstantin Boudnik added a comment - This $spark_master_url = 'yarn-client', Assume that we are always running Spark over Yarn. Which isn't the case by default. Please fix it accordingly.
        Hide
        cos Konstantin Boudnik added a comment -

        BTW, I've seen the same assumption in the 1.5.1 Spark package JIRA. Could you please address it there as well? Thanks!

        Show
        cos Konstantin Boudnik added a comment - BTW, I've seen the same assumption in the 1.5.1 Spark package JIRA. Could you please address it there as well? Thanks!
        Hide
        cos Konstantin Boudnik added a comment -

        Do we really expect anything but zeppelin-server in the future? If not, then why use long name for the role? Shall it be just 'zeppelin'?

         18   class deploy ($roles) {
         19     if ('zeppelin-server' in $roles) {
         20       include server
         21     }
         22   }
        
        Show
        cos Konstantin Boudnik added a comment - Do we really expect anything but zeppelin-server in the future? If not, then why use long name for the role? Shall it be just 'zeppelin'? 18 class deploy ($roles) { 19 if ('zeppelin-server' in $roles) { 20 include server 21 } 22 }
        Hide
        jonathak Jonathan Kelly added a comment -

        Ah ha, sorry about that. My initial patch for Zeppelin integration must not have included my changes to cluster.pp.

        And the invalid parameter "overrides" refers to something that depends upon BIGTOP-1689. This line should have been removed from this patch. Sorry about this miss too!

        Show
        jonathak Jonathan Kelly added a comment - Ah ha, sorry about that. My initial patch for Zeppelin integration must not have included my changes to cluster.pp. And the invalid parameter "overrides" refers to something that depends upon BIGTOP-1689 . This line should have been removed from this patch. Sorry about this miss too!
        Hide
        jonathak Jonathan Kelly added a comment -

        Yes, on EMR we have changed the Zeppelin UI port to 8890 because the Zeppelin default of 8080 is much too common. We have not yet suggested the change upstream in the Zeppelin project though.

        Show
        jonathak Jonathan Kelly added a comment - Yes, on EMR we have changed the Zeppelin UI port to 8890 because the Zeppelin default of 8080 is much too common. We have not yet suggested the change upstream in the Zeppelin project though.
        Hide
        jonathak Jonathan Kelly added a comment -

        What would be a more appropriate default? Spark Standalone (a spark://<master ip>:<master port> URL)? I could change the default to that.

        Show
        jonathak Jonathan Kelly added a comment - What would be a more appropriate default? Spark Standalone (a spark://<master ip>:<master port> URL)? I could change the default to that.
        Hide
        jonathak Jonathan Kelly added a comment -

        No, I don't think I expect any other roles in the future, but I named it "zeppelin-server" to match with many other roles like httpfs-server, hcatalog-server, webhcat-server, hue-server, etc.

        Show
        jonathak Jonathan Kelly added a comment - No, I don't think I expect any other roles in the future, but I named it "zeppelin-server" to match with many other roles like httpfs-server, hcatalog-server, webhcat-server, hue-server, etc.
        Hide
        cos Konstantin Boudnik added a comment -

        Yeah, something like that. Hiera provides variables for all these...

        Show
        cos Konstantin Boudnik added a comment - Yeah, something like that. Hiera provides variables for all these...
        Hide
        cos Konstantin Boudnik added a comment -

        Cool.

        Show
        cos Konstantin Boudnik added a comment - Cool.
        Hide
        cos Konstantin Boudnik added a comment -

        Let's change it here as well. This upstreaming is pretty trivial

        Show
        cos Konstantin Boudnik added a comment - Let's change it here as well. This upstreaming is pretty trivial
        Hide
        jonathak Jonathan Kelly added a comment -

        Right, not saying it would be difficult, just wondering if that's the default you'd prefer.

        Show
        jonathak Jonathan Kelly added a comment - Right, not saying it would be difficult, just wondering if that's the default you'd prefer.
        Hide
        cos Konstantin Boudnik added a comment -

        Ideally, it should be changed depends on how we deploy spark in the cluster. I believe the standalone deployment is the default for now, so yes - let's do as you've proposed. Thanks!

        Show
        cos Konstantin Boudnik added a comment - Ideally, it should be changed depends on how we deploy spark in the cluster. I believe the standalone deployment is the default for now, so yes - let's do as you've proposed. Thanks!
        Hide
        cos Konstantin Boudnik added a comment -

        Jonathan Kelly, is there any progress on this? Ideally, I would like to cut a release candidate this week. Thanks!

        Show
        cos Konstantin Boudnik added a comment - Jonathan Kelly , is there any progress on this? Ideally, I would like to cut a release candidate this week. Thanks!
        Hide
        cos Konstantin Boudnik added a comment -

        Ok, I have made expected modifications myself and pushed the patch in the master. Thanks Jonathan

        Show
        cos Konstantin Boudnik added a comment - Ok, I have made expected modifications myself and pushed the patch in the master. Thanks Jonathan
        Hide
        jonathak Jonathan Kelly added a comment -

        Konstantin Boudnik, really sorry but I have not had any time recently to work on this, and I was also on vacation for a week recently. Thanks a lot for making the necessary changes!

        Show
        jonathak Jonathan Kelly added a comment - Konstantin Boudnik , really sorry but I have not had any time recently to work on this, and I was also on vacation for a week recently. Thanks a lot for making the necessary changes!

          People

          • Assignee:
            jonathak Jonathan Kelly
            Reporter:
            cos Konstantin Boudnik
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development